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SECTION  I 
INTRODUCTION 

Computer-Based  Instruction  (CBI)  is  receiving  increasing  attention  as 
an  alternative  to  the  more  traditional  ways  of  managing  courses,  managing 
student  progress,  testing,  delivering  instruction,  and  authoring  courseware. 
The  objective  of  this  ongoing  development  is  to  find  more  effective  and/or 
efficient  means  of  accomplishing  instructional  functions.  This  requirement 
is  more  compelling  in  view  of  declining  resource  availability  for  technical 
instruction.  However,  hypothesized  efficiencies  to  be  gained  through  the 
use  of  CBI  in  a  variety  of  instructional  contexts  must  be  clearly  demon¬ 
strated  prior  to  operational  commitment  in  view  of  the  substantial  economic 
considerations  involved.  This  report  describes  one  such  demonstration.  It 
documents  the  substantial  effort  expended  in  developing  the  logic  and 
approach  used  in  the  design  and  subsequent  implementation  of  the  Computer 
Aided  Interactive  Testing  System  (CAITS)  and  addresses  the  applicability  of 
the  system  in  the  Naval  Education  and  Training  Command  ( NAVEDTRACOM) . 

BACKGROUND 

The  CAITS  is  an  automated  testing  system  designed  for  use  in  group- 
paced  instruction  with  remediation  provided  on  line  to  students  requiring 
it.  It  also  contains  an  automated  system  for  course  and  student  management. 
The  CAITS  was  originally  conceived  and  proposed  in  1980  by  the  Naval 
Education  and  Training  Program  Development  Center  Detachment  ( NAVEDTRAPRO- 
DEVCENDET),  Great  Lakes,  for  use  in  the  Airman  Apprentice  Course  at  Naval 
Training  Center,  Great  Lakes.  The  Chief  of  Naval  Education  and  Training 
(CNET)  Assistant  Chief  of  Staff  for  Training  System  Management  (N-9) 
authorized  the  development  and  pilot  test  of  the  system.  The  original  plan 
called  for  in-house  development,  implementation,  test  and  evaluation. 
Because  NAVEDTRAPRODEVCENDET,  Great  Lakes,  had  many  other  development 
commitments  during  this  period,  and  because  of  the  scope  of  the  proposed 
project,  it  became  apparent  that  the  CAITS  program  plan  of  action  and 
milestones  could  not  be  met  without  additional  resources.  Subsequently,  a 
meeting  was  held  between  representatives  from  the  Training  Analysis  and 
Evaluation  Group  (TAEG)  and  the  NAVEDTRAPRODEVCENDET,  Great  Lakes,  to 
discuss  the  proposed  development.  During  this  meeting,  it  was  suggested 
thit  TAEG  provide  support  to  NAVEDTRAPRODEVCENDET,  Great  Lakes,  in  the  areas 
of  system  design,  specification  development,  contracting,  program  monitoring 
and  evaluation. 

In  coordination  with  CNET  (N-9),  the  TAEG  was  tasked*  to  examine  alter¬ 
natives  for  accomplishing  the  proposed  development.  The  alternative  ulti¬ 
mately  selected  represented  a  mix  of  in-house  and  contract  development  with 
additional  support  provided  by  other  Navy  activities.  The  major  development 
activity  was  collocated  with  the  end  user  (schoolhouse)  thus  promoting 
greater  efficiency  in  the  development  process. 

The  special  circumstances  surrounding  this  development,  which  allowed 
the  use  of  personnel  from  a  number  of  NAVEDTRACOM  activities,  each 
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contributing  different  and  specialized  skills,  offered  a  unique  opportunity 
to  evaluate  both  this  type  of  development  effort  and  the  specific  CB I 
implementation  for  use  in  other  training  applications. 

PURPOSE 

This  report  describes  the  design,  development,  implementation,  and 
evaluation  of  the  CAITS.  It  documents  the  effectiveness/efficiency  of  CAITS 
and  provides  recommendations  for  decisions  on  employing  this  systems 
approach  in  a  variety  of  Navy  instructional  contexts. 


ORGANIZATION  OF  THE  REPORT 

In  addition  to  this  introduction,  the  report  contains  four  sections 
and  three  appendices.  Section  II  describes  the  various  aspects  of  program 
management  and  coordination  used  to  develop  the  CAITS.  It  also  summarizes 
the  system  design  approach  which  guided  all  subsequent  development 
activities.  Section  III  describes  the  desiqn  of  the  training  effectiveness 
evaluation  conducted  on  CAITS  in  the  Airman  Apprentice  course  at  Great  Lakes 
and  presents  the  analysis  of  data  collected.  Section  IV  discusses  the 
economic  feasibility  of  the  CAITS  and  provides  information  and  data  to 
support  decisions  dealing  with  the  use  of  CAITS  for  other  Navy  training 
applications.  Section  V  presents  the  conclusions  and  recommendations. 
Appendix  A  presents  the  alternative  means  considered  for  accomplishing  CAITS 
development.  Appendix  B  is  the  CAITS  functional  specification,  and  appendix 
C  is  the  CAITS  design  specification.  A  supplement  to  this  report  contains 
the  source  listings  for  all  CAITS  programs.  The  listing  is  available  from 
the  TAEG  or  the  NAVEDTRAPRODEVCEN,  Great  Lakes. 
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SECTION  II 

PROGRAM  MANAGEMENT  AND  SYSTEM  DESIGN 

This  section  describes  the  various  aspects  of  program  management  and 
coordination  used  for  the  development  of  the  CAITS.  It  documents  the  rela¬ 
tively  unique  combination  of  in-house  resources  used  to  accomplish  the 
objectives  of  the  CAITS  program.  In  addition,  it  summarizes  the  system 
design  approach  which  guided  all  subsequent  activities  of  the  development. 

PROGRAM  MANAGEMENT 

Figure  1  depicts  the  interorganizational  structure  used  to  suppc-  «;he 
CAITS  development.  The  following  paragraphs  elaborate  on  the  various  ants 
depicted  in  figure  1. 

The  concept  for  development  of  the  CAITS  was  originally  propo  'y 
education  specialists  at  the  NAVEDTRAPRODEVCENDET,  Great  Lakes.  as 
sponsored  by  the  CNET  Assistant  Chief  of  Staff  for  Training  System  Manage¬ 
ment  and  technically  supported  by  the  TAEG.  This  development  strategy 
allowed  instructional  systems  development  and  computer  program  development 
to  occur  at  a  site  collocated  with  the  using  activity  at  Great  Lakes  while 
allowing  the  system  specification  development,  program  coordination,  and 
effectiveness  evaluation  functions  to  be  accomplished  by  TAEG. 

Early  in  the  CAITS  development  program,  the  NAVEDTRAPRODEVCENDET,  Great 
Lakes,  Program  Manager  for  Airman  Apprentice  training  met  with  CNET  staff 
and  TAEG  personnel  to  suggest  a  development  strategy  for  the  CAITS  which 
would  allow  for  a  significant  level  of  system  development  prerogative  at 
Great  Lakes  without  compromising  design  goals.  It  was  proposed  that  TAEG 
personnel  assist  Great  Lakes  development  personnel  in  finalizing  a  CAITS 
design  and  then  support  the  development  and  evaluation  effort.  The  approach 
outlined  above  allowed  programs  developed  under  contract  to  be  integrated 
with  those  developed  in-house  without  excessive  delays  normally  caused  by 
travel  and  coordination  requirements.  The  onsite  development  approach  also 
enabled  close  coordination  between  the  instructional  system  designers  and 
the  computer  program  designers.  This  has  proven  to  be  a  very  important 
factor  in  the  development  process.  Subsequent  to  these  early  planning  and 
management  decisions,  TAEG  was  tasked  by  CNET  to  examine  alternatives  for 
accomplishing  CAITS  development.  TAEG  representatives  met  with 
NAVEDTRAPRODEVCENDET,  Great  Lakes,  staff  members  at  Great  Lakes  during 
February  1981  to  discuss  conceptual  alternatives  to  satisfy  CAITS 
objectives.  During  this  working  group  meeting,  a  conceptual  system  design 
was  developed.  This  design  proposed  the  use  of  an  Ohio  Scientific  classroom 
management  microcomputer  networked  to  16  Bell  and  Howell  student  station 
microcomputers.  Although  there  have  been  a  number  of  minor  hardware  and 
software  configuration  changes  since  the  conceptual  design  phase,  the 
currently  Installed  system  closely  reflects  the  original  design.  This 
Initial  working  group  system  design  was  then  used  as  a  baseline  by  TAEG 
personnel  to  evaluate  development  alternatives  for  the  implementation  of  the 
CAITS.  The  three  development  alternatives  considered  viable  for  the  time 
and  resource  constraints  imposed  included  (1)  total  development  under 
contract,  (2)  total  in-house  development,  and  (3)  a  mix  of  in-house 
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Figure  1.  CAITS  Development  Organizational  Support  Structure 
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development  and  contract  development.  This  development  alternative  assess¬ 
ment  was  submitted  to  CNET  for  consideration  on  19  February  1981  and  is 
included  in  this  report  as  appendix  A. 

After  reviewing  the  development  alternative  assessment,  CNET  tasked 
TAEG  and  NAVEDTRAPRODEVCENDET,  Great  Lakes,  to  initiate  development  for 
alternative  3  (a  mix  of  in-house  and  contractor  effort).  Under  this 
development  option,  TAEG  was  to  prepare  a  statement  of  work  and  a  contract 
package  for  computer  program  development,  develop  a  system  functional 
specification,  develop  a  hardware  and  software  system  design  specification, 
conduct  periodic  development  progress  reviews,  perform  a  system 
effectiveness  evaluation  upon  implementation,  and,  finally,  conduct  an 
economic  evaluation  for  system  cost-benefit  assessment.  The 
NAVEDTRAPRODEVCENDET,  Great  Lakes,  was  concurrently  tasked  to  procure  the 
necessary  microcomputer  equipment,  perform  the  instructional  system  design 
functions,  develop  complementary  microcomputer  programs  in  conjunction  with 
those  developed  under  contract,  perform  the  software/hardware  integration, 
arrange  for  the  required  facilities  changes,  install  the  CAITS,  and  carry 
out  all  system  implementation  tasks.  During  this  period,  NAVEDTRAPRO¬ 
DEVCENDET,  Great  Lakes,  established  a  New  Technology  Integration  Office. 
The  establishment  of  this  office  at  the  Instructional  Program  Development 
Center,  Great  Lakes,  proved  to  be  very  beneficial  during  the  CAITS  develop¬ 
ment  process.  It  provided  dedicated  resources  in  many  of  the  new  technology 
areas  for  which  policy  and  procedures  were  not  yet  available.  This 
organizational  function  should  be  considered  for  other  Instructional  Program 
Development  organizations  where  new  technology  applications  are  being 
considered. 

The  TAEG  prepared  program  planning  documents  in  coordination  with 
NAVEDTRAPRODEVCENDET,  Great  Lakes,  to  provide  a  basis  for  scheduling  and 
progress  assessment  during  program  development.  The  system  design  phase 
consisted  of  design  concept  development,  specification  development,  and 
system  design  review.  Upon  coordination  and  approval  of  the  functional 
specification  and  the  hardware/software  specification,  the  computer  program 
design  effort  was  initiated  and  purchase  orders  were  released  for  the 
computer  hardware  components.  At  each  phase  of  the  program  design,  such  as 
the  development  of  the  program  performance  specification  and  program  design 
specification,  reviews  were  conducted  to  verify  design  adequacy.  A  similar 
verification  approach  was  used  at  NAVEDTRAPRODEVCENDET,  Great  Lakes,  for  the 
development  of  the  testing  materials  to  be  entered  into  the  CAITS.  By 
following  this  modular  development  approach  and  by  testing  program  modules 
as  they  were  completed,  problems  were  identified  and  corrected  as  they 
developed.  This  resulted  in  a  minimum  of  change  during  the  final  system 
hardware/software  integration  phase.  The  courseware  and  computer  software 
development  process  was  carried  out  in  accordance  with  plan  and,  except  for 
delays  caused  by  programs  with  higher  priorities,  software/hardware 
integration  was  accomplished  on  schedule.  It  should  be  noted  that  the 
success  of  the  design  effort  relating  to  courseware,  computer  software,  and 
integration  was  due  in  large  part  to  the  expertise  of  NAVEDTRAPRODEVCENDET, 
Great  Lakes,  personnel  and  to  the  competency  of  contractor  personnel 
(University  of  Central  Florida  Contract  No.  61339-809-D-0014) .  Using  TAEG 
as  a  technical  support  agent  and  NAVEDTRAPRODEVCENDET,  Great  Lakes,  as  the 
primary  agent  for  development,  CNET  (N-9)  was  able  to  maintain  management 


11 


Technical  Report  15? 


control  without  becoming  involved  in  the  technical  detail  of  desiqn  and 
development.  This  arranqement  also  Mowed  NAVFDTRAPRODEVCFNDET,  Great 
Lakes,  to  maintain  responsibility  for  system  desiqn  and  development  while 
having  access  to  specialized  micro-computer  and  subject  matter  experts  from 
other  Navy  training  organizations. 

The  use  of  microcomputers  in  the  program  was  considered  by  CNET  and 
NAVEDTRAPRODEVCENDET,  Great  Lakes,  to  be  an  innovative  application  of  com¬ 
puters  to  test  students  and  provide  limited  instruction  in  the  form  of 
remediation.  Because  CAITS  represents  a  relatively  new  form  of  media  within 
the  NAVEDTRACOM,  it  was  necessary  to  develop  procurement,  management, 
development,  and  operations  methods  to  allow  the  integration  of  this  media 
form  into  an  ongoing  instructional  program  without  the  administrative  delays 
characteristically  associated  with  larger  scale  procurements. 

SYSTEMS  DESIGN 

This  subsection  elaborates  on  the  system  design  phase  of  the  program. 
The  material  is  presented  in  detail  since  system  design  was  the  controlling 
factor  in  all  other  aspects  of  the  development. 

The  major  phases  of  the  system  design  effort  wpre  (1)  design  concept 
development,  (2)  training  system  functional  specification  development,  (31 
system  hardware/softwa:  e  specification  development,  and  (4)  predevelopment 
training  system  desiqn  review.  Each  of  these  phases  is  described  in  detail 
in  the  following  paragraphs. 

DESIGN  CONCEPT  DEVELOPMENT.  Althouqh  the  desiqn  concept  for  the  CAITS  had 
been  outlined  prior  to  the  formation  of  the  design  and  development  team,  it 
had  not  been  subjected  to  an  in-depth  review.  Consequently,  a  number  of 
system  configuration  changes  occurred  during  the  design  concept  development 
process.  The  method  used  to  arrive  at  a  preliminary  system  concept:  was  to 
hold  a  3-day  meeting  at  the  Great  Lakes  Program  Development  Center  with 
instructional  system  design  specialists,  computer  system  design  specialists, 
and  subject  matter  experts  in  attendance.  During  this  meeting,  the  original 
CAITS  concept  was  reviewed.  This  concept  called  for  the  development  of  an 
automated  testing  and  remediation  system  for  the  4-week  course  of 
instruction  at  the  Airman  Apprentice  School.  Some  of  the  benefits  to  be 
realized  from  a  system  such  as  this  were  envisioned  to  be: 

1.  Improved  test  security  through  the  elimination  of  hard  copies, 
randomization  of  test  item  sequencing  and  response  choices,  the  use  of  test 
access  codes,  and  secured  test  authoring  stations. 

2.  Improved  student  feedback  throuoh  immediate  analysis  of  student 
response  and  instantaneous  selection  of  remedial  loops  or  follow-on  test 
sequences . 

3.  Decreased  testinq  and  remediation  lime  throuqh  immediate  response 
diaqnosis  and  remediation  selection. 
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4.  Improved  test  scoring,  student  monitoring  and  data  collection 
techniques  through  the  automation  capabilities  of  the  computer. 

5.  More  effective  evaluation  procedures  for  test  item  analysis  and 
for  the  development  of  test  validity,  reliability  and  discrimination 
measures. 

6.  Suitability  for  performance-based  testing  through  graphic  display, 
simulations,  and  job  related  scenario  displays. 

7.  Stand-alone  testing  stations  and  remote  testing  stations  with 
around-the-clock  availability  for  testing. 

8.  Automated  test  data  collection  and  reduction  with  expansion 
capability  for  automatically  communicating  student  status  information  to 
higher  command  levels. 

9.  Rapid  authoring  and  update  of  test  items  without  a  requirement  for 
printing  updated  versions  of  the  test. 

Other  potential  benefits  such  as  shipboard  traininq,  enhanced  mobiliza¬ 
tion  capability  and  adaptability  to  many  other  ratings  were  also  considered. 
The  task  of  the  system  design  group  was  to  develop  a  design  concept  which 
would  lead  to  the  realization  of  the  benefits  envisioned.  During  the 
iterative  development  of  the  design  concept,  the  system  functional 
characteristics  were  specified  and  a  number  of  hardware  and  software  system 
alternatives  were  analyzed.  Each  alternative  was  assessed  in  terms  of 
responsiveness  to  requirements,  practicality,  reliability,  supportabi 1 ity 
and  cost.  In  light  of  these  constraining  considerations,  three  con¬ 
figurations  were  selected  as  being  viable.  One  configuration  specified 
stand-alone  student  testing  stations.  A  second  configuration  called  for  a 
central  computer  networked  to  testing  terminals.  The  third  configuration, 
which  was  the  alternative  initially  selected,  is  based  on  a  microcomputer 
management  console  controlling  intelligent  student  testing  stations.  This 
configuration  allowed  for  stand-alone  operation  if  required  but  provided 
many  of  the  benefits  of  a  networked  configuration  such  as  single  node  system 
control,  automated  test  response  data  collection  and  student  station 
monitoring. 

It  was  later  decided  to  configure  the  first  installation  with  stand¬ 
alone  student  stations.  This  approach  allowed  for  evaluating  the  testing 
center  concept  before  adding  the  complexity  of  networking.  By  agreement  of 
the  working  group,  TAEG  was  to  develop  functional  and  system  design 
specifications  to  document  the  design  concept.  It  was  also  decided  that 
TAEG  would  include  related  system  requirements  such  as  operating  systems, 
data  communications,  language  options,  modular  testing  of  programs, 
documentation  and  evaluation.  The  development  of  functional  and  system 
specifications  is  discussed  below.  The  actual  specifications  which  quided 
the  development  of  CAITS  are  included  as  appendices  B  and  C. 

FUNCTIONAL  SPECIFICATION  DEVELOPMENT.  The  functional  specification 
describes  the  performance  characteristics  incorporated  into  the  design  of 
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the  CAITS.  It  includes  the  major  cateqories  of  (1)  Administrative 
Subsystem,  (2)  Test  Construction  and  Analysis,  and  (3)  Test  Delivery. 
Althouqh  the  CAITS  is  designed  as  a  testinq  system,  it  has  inherent 
capabilities  to  be  used  as  a  computer-aided  instruction  (CAIl  system, 
computer-managed  instruction  (CMI)  system,  or  instructional  administration 
system.  The  CAI  capability  is  necessary  to  deliver  remediation  which 
incorporates  text  and  graphics.  The  CMI  capability  was  necessary  to  provide 
prescriptive  feedback  to  the  student  on  either  the  computer  display  or  hard¬ 
copy  printout.  The  administrative  subsystem  is  required  as  an  extension  of 
the  testing  system  and  is  used  to  prepare  class  rosters,  testing  schedules, 
student  progress  reports,  and  test  response  summaries.  To  assure  that  these 
capabilities  would  be  included  in  the  CAITS  design,  it  was  necessary  to 
prepare  a  system  functional  specification  to  describe  system  characteristics 
in  detail  and  to  serve  as  a  design  baseline. 

The  administrative  subsystem  provides  the  functional  capability  to 
register  students,  update  student  records,  prepare  student  progress  reports, 
generate  class  rosters  and  automate  other  related  administrative  trans¬ 
actions.  The  student  test  data  contain  information  by  student  and  by  class 
on  test  identification,  scoring,  items  passed/failed,  time  on  item,  time  on 
test,  content  areas  passed/failed  and  remediation  delivered.  In  addition  to 
the  test  statistics  collected,  the  administrative  subsystem  is  designed  to 
identify  an  individual's  curriculum  track  and  to  provide  editinq  features 
for  changing  entries  if  required.  The  information  formatting  is  also 
structured  to  ensure  compatibility  with  related  CNET  training  management 
systems  such  as  CMI  and  the  Military  Personnel  Information  System 
(MILPERSIS). 

The  testing  method  selected  for  CAITS  is  based  upon  the  use  of  multiple 
choice  tests  with  up  to  50  test  items  on  each  test.  These  tests  are 
subdivided  into  content  areas  in  which  the  instructional  desiqners  can  set 
the  criteria  for  pass/fail  by  content  area  and  by  test.  At  the  completion 
of  test  delivery,  the  student  is  remediated  for  each  content  area  failed  and 
then  retested  with  alternate  test  forms  in  each  of  the  content  areas.  If 
the  student  does  not  pass  a  content  area  after  remediation,  a  prescriptive 
assignment  is  printed  out  for  off-line  review  before  re-entering  the 
training  system.  All  of  these  testing  system  functions  are  included  in  the 
functional  specification  in  accordance  with  the  CAITS  desiqn  concept. 

The  functional  description  of  the  test  construction  and  analysis 
capability  includes  such  characteristics  as  test,  item  banks,  test  item 
formats,  item  distractor  randomization  and  comprehensive  course  level  test 
generation.  It  also  includes  a  capability  for  item  analysis  for  each  test 
by  class  and  by  school.  This  capability  provides  for  statistical  treatment 
relating  to  the  analysis  of  item  and  test  difficulty,  discriminabi 1 ity, 
reliability,  and  validity. 

The  functional  requirements  for  the  test  delivery  system  address  micro¬ 
computer  hardware  and  software  capabilities  which  in  turn  must  satisfy  total 
system  functional  requirements .  For  the  pilot  implementation,  it  was 
required  that  16  students  be  accommodated  at  one  sitting  and  that  each  he 
provided  with  a  terminal  consisting  of  keyboard,  display,  and  necessary 
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curricular  materials.  It  was  also  required  that  test  security  and 
protection  features  be  built  into  the  system  and  that  a  hard-copy  backup 
system  be  maintained  to  assure  a  testing  capability  under  any  adverse 
condition  which  might  occur.  In  addition,  the  functional  design  provides 
for  future  expansion  of  testing  capabilities,  such  as  the  addition  of 
constructed  response  features.  As  a  final  functional  requirement,  it  was 
specified  that  student  stations  be  capable  of  operating  in  remote  stand¬ 
alone  configuration  or  in  an  integrated  network  mode  with  other  host 
computers.  The  functional  specification  which  contains  the  requirements 
described  herein  is  included  as  appendix  B. 

HARDWARE  AND  SOFTWARE  SPECIFICATION  DEVELOPMENT.  The  TAEG  initiated 
development  of  the  system  harJware  and  software  specification  upon 
completion  of  the  system  functional  specification.  This  document  specified 
computer  hardware  components,  computer  programs,  system  hardware/software 
integration,  documentation  requirements,  and  system  test  standards. 
Although  the  CAITS  is  primarily  an  in-house  small  scale  development,  it 
utilized  relatively  complex  microcomputer  hardware  and  software. 
Consequently,  it  was  necessary  to  follow  standard  design  and  development 
procedures.  In  this  system,  a  number  of  microcomputers  are  utilized  as 
components  of  a  classroom  testing  center  in  which  computer  programs  and 
courseware  are  integrated  to  provide  the  needed  operational  capability.  To 
reduce  development  risk,  a  number  of  microcomputer  products  were  surveyed. 
Operating  systems  and  languages  were  evaluated  and  special  purpose  hardware 
and  software  components  were  tested  in  house  before  making  design  decisions. 
The  final  conf iguration  selected  for  the  CAITS  consisted  of  an  Ohio 
Scientific  microcomputer  with  a  10  megabyte  hard  disk  to  be  utilized  as  a 
classroom  management  station.  This  was  later  changed  to  a  Bell  and  Howell 
system  to  allow  for  interchangeability  with  student  stations.  This  station 
was  also  to  have  a  capability  for  authoring,  evaluation,  and  administrative 
support.  Initial  designs,  which  were  later  rejected,  included  high  speed 
parallel  data  transfer  between  the  host  and  the  student  terminals  to 
eliminate  the  requirement  for  floppy  disks  at  the  student  station.  This 
design  approach  was  rejected  because  it  did  not  allow  for  stand-alone 
student  station  operation.  It  should  be  mentioned,  however,  that  follow-on 
configurations  most  likely  will  include  high  speed  serial  data  interchange 
to  provide  data  transmission  to  and  from  the  host  computer.  A  network 
configuration  is  also  envisioned  with  64  or  128K  of  RAM  storage  at  each 
student  station.  A  number  of  off-the-shelf  local  networking  packages  were 
also  considered  but  were  not  included  in  the  initial  design  because  of  cost 
and  increased  development  risk. 

A  major  issue  during  the  survey  was  that  of  language  selection.  The 
candidate  approaches  included  the  use  of  Apple  PILOT,  UCSD  PASCAL,  compiled 
BASIC  and  standard  interpreted  BASIC.  PILOT  was  rejected  because  of  the 
desire  to  have  complete  development  flexibility  without  having  to  configure 
student  stations  for  a  PASCAL-based  language  such  as  the  Apple/Bell  and 
Howell  PILOT.  PASCAL  was  rejected  for  similar  reasons  since  a  language  card 
extension  to  the  48K  Apple  main  memory  would  be  required  at  each  student 
station.  A  disk-based  language  also  adds  complexity,  in  comparison  to  the 
ROM-based  applesoft  BASIC  interpreter,  and  was  not  considered  desirable  for 
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student  stations.  The  compiled  versions  of  BASIC  were  also  rejected  because 
of  added  complexity  without  any  significant  offsetting  benefit.  After  con¬ 
siderable  discussion,  the  design  group  developed  an  appreciation  for  the 
advantages  of  interpreted  BASIC  as  a  development  language  and  better  under¬ 
stood  the  reasons  for  its  emergence  as  the  leading  personal  computer 
language.  It  was  selected  as  the  development  language  for  the  CAITS  and  as 
of  this  time  it  is  a  decision  which  still  appears  to  be  supportable.  BASIC 
is  a  relatively  simple  language,  yet  possesses  the  power  needed  for  many 
applications  of  this  type.  If  greater  computational  speed  and/or  language 
capabilities  are  required  on  future  applications,  such  as  simulation-based 
training,  it  is  evident  that  other  operating  systems,  languages,  and  hard¬ 
ware  options  will  have  to  be  considered.  However,  the  design  concept  of  not 
using  more  computational  power  than  is  reasonably  needed  continues  to  be  one 
supported  by  the  CAITS  design  group. 

Documentation  requirements  for  this  development  included  a  program  per¬ 
formance  specification,  program  design  specification,  program  listings,  test 
procedures  and  a  user's  manual.  Although  this  limited  amount  of  documenta¬ 
tion  would  be  considered  minimal  and  possibly  inadequate  for  a  major  train¬ 
ing  system  development,  it  has  proven  to  be  more  than  adequate  for  the 
CAITS.  Because  8ASIC  program  listings  are  easily  understandable  by  qual¬ 
ified  BASIC  programmers,  there  have  been  no  major  problems  in  maintaining 
configuration  control  on  CAITS  programs.  If  good  programming  practices  are 
followed,  such  as  liberal  use  of  remarks,  minimal  use  of  multi -statement 
lines,  and  modularization  of  programs,  program  update  and  maintenance  can  be 
easily  and  effectively  accomplished.  System  test  and  evaluation  were  also 
addressed  in  the  hardware/software  specification  and  system  tests  were  con¬ 
ducted  prior  to  courseware  integration. 

SYSTEM  DESIGN  ENHANCEMENTS 

During  the  CAITS  development  and  preliminary  testing  phase,  a  number  of 
potential  enhancements  were  identified.  These  enhancements  were  necessary 
to  correct  deficiencies  noted  during  schoolhouse  validation  of  software.  A 
progam  update,  which  was  accomplished  at  NAVEDTRAPRODEVCENDET,  Great  Lakes, 
to  incorporate  these  enhancements,  is  identified  as  CAITS  version  2.0.  It 
is  the  current  version  of  the  CAITS  software.  The  enhancements  incorporated 
provide  the  following  functional  capabilities: 

1.  Allow  students  the  ability  to  change  answers  while  an  item  is  dis¬ 
played  on  the  screen  (however,  maintain  the  restriction  that  once  the  stu¬ 
dent  goes  to  the  next  item  he/she  cannot  return  to  it  to  change  answers). 

2.  Permit  students  the  option  of  passing  over  an  item  to  return  to  it 
later  (when  no  answer  is  entered). 

3.  Display  the  total  number  of  questions  and  the  number  of  items 
remaining  on  the  test  at  the  bottom  of  each  screen. 

4.  Plan  for  interruptions  during  the  test  to  allow  for  re-entry  at 
the  point  interrupted. 
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5.  Allow  for  instructor  specified  remediation  and  retesting  in  a 
CAITS  override  mode. 

6.  Allow  for  latent  remediation  and  retesting.  With  this  capability 
students  can  return  for  remediation  and  retesting  if  authorized  by  the 
Instructor. 

7.  Integrate  manual  testing  methods  with  automated  methods.  Under 
version  2.0  testing  can  be  accomplished  with  hard-copy  tests  and  optically 
scanned  answer  sheets  as  well  as  by  microcomputer.  Testing  under  both 
methods  can  be  concurrent  and  answer  sheet  data  can  be  merged  with  floppy 
disk  data  for  full  CAITS  management  report  generation. 

8.  Provide  additional  reporting  enhancements  including  cumulative 
scoring,  class  summary  statistics,  student  ranking,  students  remediated, 
class  rosters,  student  transfers/drop  list,  and  automated  Academic  Review 
Board  (ARB)  worksheets. 

9.  Provide  a  file  editor  program  to  allow  for  easy  data  entry,  edit¬ 
ing  or  update.  With  this  feature,  item  banks  can  be  reviewed  and  modified, 
if  necessary,  by  school  management.  Student  scores  from  classes  which  test 
with  paper  and  pencil  can  also  be  merged  with  the  computer  generated  data 
base  for  subsequent  reporting  and  management  purposes. 

10.  Provide  randomization  of  test  item  sequencing  and  response 
choices. 

All  of  these  program  features  have  been  tested  and  validated  and  are 
currently  being  utilized  for  Airman  Apprentice  testing.  The  implementation 
of  the  total  CAITS  required  only  the  installation  of  off-the-shelf  micro¬ 
computer  hardware  and  the  integration  with  the  CAITS  program  package  as 
described  herein.  The  expansion  of  this  system  or  the  installation  of  a 
system  at  another  location  would  require  only  the  addition  of  hardware  and 
the  integration  with  existing  software.  Only  test  item  entry  would  be 
required  for  other  testing  applications.  Since  test  item  generation  is  nec¬ 
essary  for  testing  in  any  mode,  this  effort  would  not  normally  be  considered 
a  cost  for  new  applications.  It  is  also  probable  that  the  efficiencies 
gained  by  having  machine  readable  test  item  data  in  test  data  banks  would 
result  in  lower  overall  cost  for  test  generation  and  update.  The  hardware 
system  as  currently  configured  is  shown  in  figure  2. 

The  software  for  this  system  is  contained  on  the  floppy  disks  in  the 
form  of  an  instructor  disk,  a  student  disk,  and  an  authoring  disk.  The 
instructor  disk  contains  the  CAITS  management  programs  and  is  menu  driven. 
The  student  disk  contains  student  test  delivery  and  remediation  programs. 
The  authoring  disk  contains  the  programs  needed  to  create  and  revise  items 
and  remedial  data  banks.  Two  complete  tests  with  remediation  can  be  stored 
on  each  side  of  a  floppy  disk  for  the  Airman  Apprentice  Program. 
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SECTION  III 

TRAINING  EFFECTIVENESS  EVALUATION 

This  section  describes  the  design  of  the  training  effectiveness  evalua¬ 
tion  (TEE)  of  CAITS  in  the  Airman  Apprentice  course  at  Great  Lakes  and 
presents  the  analyses  of  data  collected. 

The  TEE  of  CAITS  addressed  the: 

•  effect  of  the  computer-presented  versus  traditional  paper  and 
pencil  testings  on  the  amount  of  time  spent  testing 

•  effect  of  computer  tests  versus  paper  and  pencil  tests  on  student 
performance  (i.e.,  test  results) 

•  effect  on  total  testing  time  and  student  performance  of  remedia¬ 
tion  provided  by  CAITS  compared  to  that  traditionally  provided  by 
instructors. 

AIRMAN  APPRENTICE  COURSE  TESTING 

The  Airman  Apprentice  course  is  a  4-week,  group-paced  course  which  has 
been  developed  in  conformance  with  the  Instructional  Systems  Development 
procedures  outlined  in  NAVEDTRA  110A  and  CNETINST  1550.15. 

There  are  currently  10  end-of -module,  criterion-referenced  tests  which 
occur  following  logically  complete  and  related  segments  of  instruction. 
These  tests  contain  from  20  to  50  multiple  choice  items  (over  90  percent  of 
the  items  are  multiple  choice;  the  remainder  are  true-false  and  matching) 
which  are  administered  to  all  students  following  the  segment  of  instruction. 
The  basic  purpose  of  the  tests  is  to  determine  whether  or  not  each  student 
has  achieved  the  behavioral  objectives  which  are  the  bases  of  the 
instructional  material  and  test  items. 

DESCRIPTION  OF  TESTS,  TEST  SEQUENCE,  AND  REMEDIATION 

The  current  Airman  Apprentice  course  test  items  were  used  in  this 
evaluation.  They  were  the  same  for  both  computer  presentation  and  for  the 
traditional  paper  and  pencil  testing.  For  those  items  which  require 
illustrations,  a  supplemental  sheet  of  illustrations  was  provided  to 

accompany  the  computer  presentation  of  the  items.  A  prompt  appeared  on  the 
monitor  directing  the  student's  attention  to  the  appropriate  place  on  the 

supplemental  sheet.  (In  the  future,  it  will  be  possible  to  exploit  the 
graphics  capabilities  of  the  computer.) 

The  sequence  and  frequency  of  testing  conformed  with  that  of  the 

present  Airman  Apprentice  course.  The  10  testing  points  occurred  following 
related,  logically  cohesive  segments  of  instruction.  Each  such  segment 

contained  from  one  to  four  terminal  objectives  developed  in  accordance  with 
the  ISD  model.  Testing  in  the  computer  and  paper-and-pencil  modes  occurred 
at  the  same  points  in  the  instructional  sequence  and  covered  the  same 
terminal  objectives. 
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Remediation  procedures  on  CAITS  differed  from  current  procedures. 
Currently,  remediation  consists  of  the  instructor  reviewing  frequently 
missed  items  directly  following  scoring  of  the  test  and  "night  study"  for 
those  students  who  did  not  meet  criterion  on  the  initial  testing.  "Night 
study"  means  that  the  student  rereads  all  printed  curricular  material 
associated  with  the  lesson,  asks  the  instructor  for  clarification  of  any 
confusing  area,  and  then  retakes  an  alternate  form  of  the  end-of -module 
test.  This  remediation  is  a  nonspecific  "broad  brush"  approach  which 
addresses  sets  of  content  areas  associated  with  as  many  as  four  terminal 
objectives.  Since  the  computer-based  testing  affords  the  opportunity  for 
"on-line"  remediation,  it  was  decided  to  incorporate  remediation  based  on 
more  discrete,  and  thus  more  precise,  diagnosis.  Following  each  test 
administration,  the  computer  evaluated  the  examinees'  responses  and  provided 
a  pass/fail  report  for  the  test.  If  a  student  did  not  attain  the  criterion 
cut-score,  remediation  was  given  immediately  by  CAITS.  Following  remedia¬ 
tion,  a  retest  was  given,  consisting  of  test  items  from  the  current 
alternate  form  of  the  test. 

DATA  ANALYSES 

The  data  from  this  study  were  analyzed  in  three  parts.  Eight  variables 
used  in  the  analyses  were: 

•  comprehensive  test  score  (raw  score) 

•  test  scores  (percent  correct) 

•  Armed  Forces  Qualification  Test  (AFQT)  score 

•  years  of  education  (number  of  years) 

•  Reading  score  ( Gates -MacGini tie  Reading  Test) 

•  Word  Knowledge  (subtest  of  the  Armed  Services  Vocational  Aptitude 
Battery) 

•  testing  times  (active  time  on  the  computer) 

•  qroup  assignment. 

In  the  first  part,  a  multiple  reqression/correlation  analysis  was  conducted 
on  comprehensive  test  scores  using  various  background  information  (AFQT 
score,  years  of  education,  reading  score,  word  knowledge)  and  group 
assignment.  This  analysis  was  performed  to  examine  possible  differences  in 
comprehensive  test  scores  amonq  three  treatment  groups  while  controlling  for 
the  possible  effects  of  background  variables.  In  the  second  part,  the 
effects  of  remediation  on  test  scores  were  assessed  by  t-test  analyses.  In 
the  third  part,  averages  were  computed  of  testing  times  and  remediation 
times  for  determining  the  dui  ation  of  testing  sessions.  A  description  of 
the  sample  and  the  results  from  the  three  parts  of  analyses  are  described 
below.  Questionnaires  were  administered  to  the  two  experimental  groups  that 
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were  exposed,  in  part  or  in  full,  to  the  computer-presented  treatment  in 
order  to  identify  student  attitudes  toward  CAITS. 

SAMPLE.  The  training  effectiveness  evaluation  is  based  on  a  comparative 
analysis  of  three  groups.  Subjects  in  the  study  were  randomly  assigned  to 
one  of  these  groups: 

•  experimental  group  with  on-line  remediation 

•  experimental  group  with  class  review  of  the  test 

•  control  group. 

The  experimental  group  with  remediation  (n=92)2  comprised  those 
students  who  were  tested  on  the  computer  and  received  remediation  when  their 
test  scores  on  a  content  area  fell  below  a  specified  criterion.  Students  in 
the  experimental  group  without  remediation  ( n=103 )  were  also  tested  on  the 
computer.  However,  regardless  of  their  score  on  a  content  area,  they  were 
given  class  review  of  the  test.  The  control  group  (n=101)  was  tested  by 
paper-and-pencil  tests  and  given  class  review. 

DIFFERENCES  AMONG  TREATMENT  GROUPS  ON  COMPREHENSIVE  TEST  SCORES.  Multiple 
regression/correlation  analysis  was  performed  on  the  comprehensive  test 
scores  of  all  three  groups  to  determine  differences  among  the  three 
treatment  groups  in  comprehensive  test  scores  while  controlling  for  AFQT 
scores,  years  of  education,  reading  score,  word  knowledge,  and  group 
assignment.  Means  and  standard  deviations  for  the  variables  used  in  this 
analysis  are  presented  in  table  1. 

The  two  experimental  group  variables  are  analyzed  in  this  regression 
model  using  the  dummy  variable  technique. 3  This  technique  involves  the 
creation  of  dummy  variables  that  reflect  group  membership.  In  this  case, 
since  there  are  three  groups,  one  control  and  two  experimental,  two  dummy 
variables  are  required.  One  dummy  variable  is  coded  with  a  value  of  "1"  if 
the  student  is  a  member  of  the  first  experimental  group  and  "0"  if  he  or  she 
is  not.  The  other  dummy  variable  is  coded  "1"  if  the  student  is  a  member  of 
the  second  experimental  group,  and  "0"  otherwise.  If  a  student  has  "Os"  on 
both  dummy  variables,  then  he  or  she  must  be  a  member  of  the  control  group. 
Thus,  only  two  dummy  variables  are  necessary  to  analyze  the  differences 
among  three  groups. 


^Thirteen  students  assigned  to  the  experimental  group  with  remediation,  for 
various  reasons,  aborted  the  testing  sessions  resulting  in  inconclusive 
remediation  score  data;  therefore,  analyses  for  determining  the  effects  of 
remediation  included  only  79  students. 

3For  a  more  detailed  explanation  of  the  dummy  variable  technique,  see 
J.  Cohen  and  P.  Cohen.  Applied  Multiple  Regression/Correlation  Analysis 
for  the  Behavioral  Sciences.  Hillsdale.  N.J.:  Lawrence  Erlbaum 
Associates,  1$)5. 
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Based  on  the  results  of  the  regression  analysis  reported  in  table  2, 
there  are  no  significant  differences  among  the  three  treatment  groups  in 
comprehensive  test  score,  controlling  for  AFQT  score,  years  of  education, 
reading  score,  and  word  knowledge  score.  In  fact,  the  two  experimental 
groups  scored  slightly  higher  than  the  control  group  (table  1).  Therefore, 
at  the  very  least,  the  CAITS  resulted  in  performance  equivalent  to  that 
obtained  with  more  traditional  methods. 


TABLE  2.  MULTIPLE  REGRESSION  OF  COMPREHENSIVE  TEST  SCORE  ON 
TREATMENT  GROUP,  STANDARDIZED  TEST  SCORES,  AND 
YEARS  OF  EDUCATION 


Independent  Variables 

Regression  Coefficient 

t 

Experimental  Group  1 

1.47 

1.53 

Experimental  Group  2 

1.51 

1.72 

AFQT 

.06 

2.20* 

Years  of  Education 

1.01 

2.74* 

Reading  Score 

.71 

2.97* 

Word  Knowledge 

.11 

1.38 

Note:  R2  =  .20,  F  (6,239)  =  9.66,  £<.001. 
*  indicates  significant  t-test  with  £<.05. 


EFFECTS  OF  REMEDIATION  ON  TEST  SCORES.  Ten  tests  comprised  of  39  content 
areas  were  administered  to  the  students.  For  those  students  within  the 
experimental  group  with  on-line  remediation,  each  content  area  had  a 
potential  for  remediation  whenever  the  student's  number  of  correct  responses 
was  lower  than  a  specified  criterion.  Sixty-nine  of  the  seventy-nine 
students  within  the  experimental  group  with  on-line  remediation  were 
remediated.  The  minimum  number  of  remediations  by  a  remediated  student  was 
one;  the  maximum  was  nine.  For  all  the  students  who  were  remediated,  an 
average  of  3.57  remediations  occurred.  For  the  246  occurrences  of 

remediation,  there  were  only  155  instances  (63  percent)  where  the  post¬ 
remediation  test  score  was  adequate  for  passing  a  particular  content  area. 

To  examine  the  effects  of  remediation,  t-tests  were  performed  on  the 
mean  content  area  pre-  and  post -remediation  test  difference  scores  and  on 
the  mean  content  area  comprehensive  test  scores  for  which  remediation  was 
indicated. 
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For  all  the  content  areas  in  which  remediation  occurred,  averages  of 
the  pre-remediation  test  scores  and  averages  of  the  corresponding  post¬ 
remediation  test  scores  were  computed  to  give  average  pre-remediation  and 
average  post-remediation  test  scores,  respectively.  To  determine  the 
immediate  effects  of  remediation  on  test  scores,  the  difference  between  the 
pre-and  post-remediation  test  scores  was  tested  for  significant  difference 
from  zero.  The  results  of  the  t-test  showed  a  significant  difference 
between  the  average  pre-  and  post-remediation  test  scores  (t  =  11.31,  df  = 
68).  The  average  post-remediation  test  scores  were  higher  (mean  difference 
=  21.62),  indicating  that  remediation  did  improve  a  student's  test  score  in 
the  short  term. 

To  determine  the  long-term  effects  of  remediation  on  test  scores,  the 
mean  content  area  test  scores  and  the  mean  content  area  comprehensive  test 
scores  were  compared  between  those  students  who  were  remediated  (n=69)  and 
those  students  who  would  have  received  on-line  remediation  but  did  not 
because  of  being  assigned  to  the  experimental  group  with  class  review  of  the 
test  (n=84).  Mean  content  area  test  scores  and  mean  content  area  comprehen¬ 
sive  test  scores  were  computed  by  averaging  the  test  scores  and  the 
comprehensive  test  scores,  respectively,  for  all  the  content  areas  where  the 
number  of  correct  responses  was  lower  than  the  specified  criterion.  For  the 
two  groups,  the  results  of  the  t-test  showed  that  there  were  no  significant 
differences  in  the  mean  content  area  test  scores  (t  =  0.44,  df  =  151)  or  in 
the  mean  content  area  comprehensive  test  scores  (t  =  0.18,  df  =  93) 
therefore,  remediation  did  not  have  a  significant  long-term  effect  on 
improving  a  student's  comprehensive  test  score, 

DURATION  OF  TESTING  SESSION.  To  determine  the  duration  of  the  testing 
sessions,  averages  were  computed  of  the  testing  times  for  each  test  for  the 
two  experimental  groups  that  were  tested  by  the  computer.  Table  3  presents 
the  averages. 

As  can  be  observed  from  table  3,  test  #7  had  the  shortest  average 

testing  time,  while  test  #6  had  the  longest  average  testing  time.  The 

average  testing  time  considering  all  testing  sessions  was  28.51  minutes. 

This  compares  to  1  hour  for  each  testing  session  for  the  control  group. 

In  addition  to  the  above  testing  time,  students  in  the  experimental 

group  with  on-line  remediation  also  spend  time  in  remediation.  Therefore, 
for  each  test,  averages  were  computed  of  the  amount  of  time  students  were  in 
remediation.  These  average  remediation  times  are  reported  in  table  4.  The 
average  time  a  student  spends  in  remediation  for  any  of  the  tests  was  less 
than  5  minutes. 


^Content  area  comprehensive  test  scores  were  missing  for  58  cases.  Thus, 
t-tests  were  performed  to  determine  bias  between  the  two  groups  due  to 
missing  comprehensive  test  scores.  No  significant  differences  in  average 
test  scores  were  found. 
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TABLE  3. 

AVERAGE  TESTING  TIMES 

FOR  EXPERIMENTAL  GROUPS 

Mean 

Standard  Deviation 

N 

Test 

#1 

30.83* 

7.39 

194 

Test 

#2 

24.55 

5.83 

191 

Test 

#3 

31.26 

7.15 

189 

Test 

#4 

22.37 

5.41 

189 

Test 

#5 

35.66 

9.12 

185 

Test 

#6 

38.44 

9.00 

180 

Test 

#7 

18.87 

5.34 

180 

Test 

#8 

21.72 

5.76 

177 

Test 

#9 

31.25 

6.39 

174 

Test 

#10 

30.18 

6.19 

173 

Over 

All 

Tests 

28.51 

9.15 

195 

♦Time  is  in  minutes. 
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TABLE  4.  MEAN  REMEDIATION  TIMES 


Mean 

Standard  Deviation 

N 

Test  #1 

4.83* 

3.18 

26 

Test  #2 

3.29 

1.99 

25 

Test  #3 

2.45 

0.88 

8 

Test  #4 

4.76 

2.64 

10 

Test  #5 

3.43 

2.17 

43 

Test  #6 

3.87 

3.88 

7 

Test  #7 

2.95 

2.17 

11 

Test  #8 

3.57 

2.19 

26 

Test  #9 

2.42 

1.11 

18 

Test  #10 

1.96 

1.86 
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STUDENT  ATTITUDES  TOWARDS  CAITS.  Questionnaires  were  administered  to  the 
two  experimental  groups  which  were  exposed,  in  part  or  in  full,  to  the 
computer  presented  treatment.  The  students  were  asked: 

•  How  well  do  you  think  the  tests  tested  your  knowledge? 

•  Do  you  think  the  tests  were  difficult? 

•  What  is  your  feeling  about  the  way  the  tests  were  given? 

Average  responses  to  the  above  questions  were  computed  for  each  class 
and  over  all  the  classes.  Overall  means  and  frequencies  of  responses  are 
presented  in  table  5.  Students  thought  the  tests  tested  their  knowledge 
fairly  to  very  well,  the  tests  ranged  from  slightly  to  somewhat  difficult, 
and  their  feelings  were  neutral  to  positive  about  the  way  the  tests  were 
given . 


?1 


OVERALL  MEANS  AND  FREQUENCIES  OF  STUDENT  RESPONSES  TO  QUESTIONNAIRE 


1  Standard  Deviation 
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SECTION  IV 
ECONOMIC  ANALYSIS 

This  section  discusses  the  economic  feasibility  of  the  CAITS  and 
provides  information  and  data  to  support  decisions  dealing  with  extending 
the  use  of  CAITS  to  other  Naval  training  applications.  The  analysis  and 
cost  data  presented  in  this  section  are  based  on  the  following  assumptions: 

1.  The  planning  period  is  limited  to  7  years  because  it  is  assumed 

that  improvements  in  microcomputer  technology  within  this  period  will  make 
current  technology  economically  obsolete. 

2.  The  salvage  value  of  all  hardware  is  estimated  at  10  percent  of 

the  original  purchase  price. 

3.  The  costs  and  benefits  are  discounted  at  a  rate  of  10  percent. 

4.  The  costs  are  considered  incremental. 

5.  Additional  instructor  and  administrative  support  will  be  required 
to  install  additional  CAIT  systems. 

6.  Administrative  and  instructor  support  will  be  required  to  operate 
additional  systems  but  because  of  a  restructuring  of  the  work  load  this 
support  can  be  provided  from  resources  already  available  and  will  not 
require  new  or  additional  resources. 

7.  The  only  additional  resources  required  to  operate  the  system  on  an 
annual  basis  will  be  for  maintenance  of  equipment  and  supplies. 

COST  ANALYSIS 

Since  initiation  of  the  CAITS  program  in  1980,  there  has  been  some 
improvement  in  the  capability  of  microcomputers  and  a  continuation  of  the 
downward  trend  in  hardware  prices.  Consequently,  the  hardware  costs 
experienced  in  acquiring  the  pilot  system  are  no  longer  relevant  for 
estimating  the  costs  of  additional  hardware  for  expansion  or  for  other 
applications.  The  following  cost  analysis  is  divided  into  three  parts.  The 
first  part  documents  the  actual  costs  of  conducting  the  pilot  program 
including  both  the  hardware  and  development  costs.  The  second  part  provides 
estimates  of  the  costs  for  continuing  to  operate  the  15  student  station 
system  developed  for  the  pilot  program.  The  third  part  provides  an  estimate 
of  the  life-cycle  costs  of  expanding  the  CAITS.  All  costs  are  based  on  a 
CAITS  consisting  of  15  student  testing  stations  and  one  instructor  station 
with  minimum  equipment  to  serve  as  backup. 

COST  OF  THE  PILOT  SYSTEM.  Each  student  station  includes  one  Apple  II  micro¬ 
computer,  one  single  disk  drive,  and  one  monitor.  The  instructor  station 
includes  one  Apple  II  microcomputer,  two  disk  drives,  a  monitor,  a  printer, 
and  a  clock  card.  The  total  costs  of  purchasing  and  installing  the  hard¬ 
ware,  developing  the  initial  software  and  providing  the  in-house  support  for 
acquiring  and  installing  the  CAITS  are  presented  in  table  6.  The  total 
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Includes  one  instructor  station  and  spare  equipment. 
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one  time  investment  costs  amounted  to  $88,510.  The  estimated  annual  cost  of 
operating  the  system  after  it  has  been  developed  and  implemented  is  $500  per 
year  for  supplies  (which  are  primarily  disks  and  paper  for  printing  student 
reports)  and  $1,000  for  maintenance  of  the  equipment.  The  net  present  cost 
of  the  pilot  program  is  estimated  at  $94,022  for  the  7-year  period. 
Potential  cost  savings  which  might  result  from  the  elimination  of  the  paper 
used  for  the  present  testing  system  are  considered  minimal  and  are  not 
included. 

RETENTION  COST  OF  PILOT  SYSTEM.  The  investment  iri  the  software  and  support¬ 
ing  hardware  for  the  pilot  system  has  been  committed  to  the  program  and  is 
in  fact  a  sunk  cost.  Therefore,  cost  is  irrevelant  to  the  decision  of 
whether  it  is  economically  feasible  to  continue  with  the  pilot  system  in  the 
operational  environment.  The  current  value  of  the  resources  invested  in  the 
system  (i.e.,  the  costs  which  are  currently  recoverable)  is  estimated  at  50 
percent  of  the  hardware  costs.  The  recoverable  costs  are  estimated  at 
$22,005.  It  is  assumed  that  none  of  the  investment  in  software  and 
installation  costs  have  an  alternative  value.  The  life-cycle  cost 
(discounted  at  10  percent)  to  continue  the  operation  of  the  15  student 
station  system  acquired  for  the  pilot  program  is  estimated  at  $28,595  or 
approximately  $4,000  per  year  (table  7).  However,  the  incremental  operating 
costs  estimated  at  $1,500  per  year,  are  the  only  additional  funds  which  must 
be  provided  to  continue  the  operation  of  the  pilot  system. 


TABLE  7.  ESTIMATED  LIFE-CYCLE  COSTS  OF  CONTINUED  UTILIZATION  OF  THE 
COMPUTER  AIDED  TESTING  SYSTEM  ACQUIRED  IN  PILOT  PROGRAM 


Resource 

Current 

Equip 

Value 

1 

2 

3 

Year 

4 

5 

6 

7 

Salvage 

Value 

Equipment 

22005 

220.5 

maintenance 

1000 

1000 

1000 

1000 

1000 

1000 

1000 

Suppl ies 

500 

500 

500 

500 

500 

500 

500 

Labor 

contracted 

0 

0 

0 

0 

0 

0 

0 

in-house 

0 

0 

0 

0 

0 

0 

0 

Total 

22005 

1500 

1500 

1500 

1500 

1500 

1500 

1500 

220.5 

Net  present 

value  (28,595) 
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LIFE-CYCLE  REPLICATION  COSTS.  Further  expansion  of  the  CAITS  will  be  less 
expensive  than  the  costs  incurred  for  the  establishment  of  the  pilot  system. 
Two  factors  contribute  to  the  lower  costs.  First,  the  cost  of  developing 
the  generic  software  need  not  be  duplicated  as  additional  systems  are 
brought  on-line.  The  cost  of  any  generic  software  modifications  necessary 
for  expansion  of  the  system  are  minimal  or  nonexistent.  Second,  there  has 
been  a  decrease  in  the  cost  of  microcomputer  hardware  since  the  hardware  was 
acquired  for  the  pilot  program.  The  estimated  life-cycle  cost  for  each 
additional  15  student  station  testing  system  is  presented  in  table  8.  The 
estimated  life-cycle  cost  (discounted  at  10  percent)  is  estimated  at  $54,097 
or  approximately  $7,700  per  year.  The  initial  investment  costs  are 
estimated  at  $48,100  with  annual  operating  costs  estimated  at  $1,500  per 
year.  A  summary  of  all  three  aspects  of  the  cost  analysis  is  presented  in 
table  9.  A  CAITS-type  testing  system  can  be  developed  and  implemented  for 
other  lock-step  courses,  but  there  would  be  additional  resource  requirements 
for  the  development  of  the  test  items  and  remediation  modules  which  are 
relevant  to  the  course.  A  unique  economic  analysis  would  be  required  to 
support  decisions  about  specific  applications. 

BENEFITS 

The  CAITS  has  the  capability  to  replace  pencil  and  paper  tests  and 
provide  immediate  remediation  to  those  students  who  require  it.  In 
addition,  CAITS  provides  a  capability  for  selective  remediation  with  respect 
to  course  content.  The  current  testing  system  requires  a  1-hour  testing 
period  followed  by  a  1-hour  block  of  time  for  review  and  remediation.  Data 
from  the  CAITS  pilot  study  show  that  those  students  who  were  remediated, 
with  the  computer,  required  an  average  of  less  than  5  minutes  of  selective 
remediation  per  test.  Students  were  remediated  only  in  content  areas  in 
which  a  performance  deficiency  was  demonstrated. 

The  computer  aided  testing  time  per  student  per  test  was  approximately 
one-half  hour  as  reported  in  section  III.  The  total  estimated  time, 
including  remediation  and  overhead  time,  was  estimated  at  .55  hour  per  test 
per  student.  The  total  estimated  time  savings  over  the  testing,  review  and 
remediation  time  required  for  p-esent  methods  was  estimated  at  1.45  hours 
per  student  per  test.  Table  1U  summarizes  the  test  and  remediation  times 
for  both  the  CAITS  and  the  conventional  pencil  and  paper  testing  methods, 
assuming  that  one  15  student  station  testing  center  is  used. 

A  15  student  station  CAITS  can  accommodate  1,000  students  per  year.  It 
is  estimated  that  the  acquisition  and  use  of  such  a  system  could  save  14,500 
student  training  hours  per  year  (table  11). 
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TABLE  9.  SUMMARY  TABLE  OF  COSTS  OF  IMPLEMENTING  A  15  STUDENT 
STATION  COMPUTER  AIDED  INTERACTIVE  TESTING  SYSTEM 


Alternative 

Life-Cycle  Costs* 

Total  Average  Annual 

Annual 

Operating 

Costs 

Pilot  System 

$94,022 

$13,432 

$1 , 500 

Retention  of 

Pilot  System 

28,595** 

4,085 

1,500 

System  Replication 

54,097 

♦Based  on  present  value 
♦♦Excludes  sunk  costs. 

costs  computed  using  a  10  percent 

discount  rate. 

TABLE  10.  AVERAGE  STUDENT  TEST  AND 
FOR  CAITS  AND  TRADITIONAL 

REMEDIATION  TIME 
TESTING 

PER  TEST 

Testing 

Testing 

Posttest 

System 

Time 

Remediation 

Total 

(Hours) 

CAITS 

.55* 

None 

.55 

Paper  Tests 

1.00 

1.00 

2.00 

Time  Reduction 

1.45 

*  Includes  remediation 

and  overhead  time. 

TABLE  11.  ESTIMATED  POTENTIAL  TIME  SAVINGS  PER  YEAR  FOR  A  15  STUDENT 
STATION  TEST  CENTER  WITH  1,000  STUDENT  THROUGHPUT 


Number  Tests  Per  Time  Reduction  Total  Time 

Students  Student  Test  Student  Savings 

( Hours) 
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Approximately  2,000  students  per  year  complete  the  Airman  Apprentice 
course  at  Great  Lakes.  If  the  CAITS  were  expanded  for  that  course  so  that 
all  student  testing  could  be  done  on  the  computer,  then  an  estimated  29,000 
testing  hours  could  be  eliminated  from  the  instructional  progam.  However, 
an  additional  15  student  station  system  would  be  required  to  service  the 
entire  Airman  Apprentice  course.  The  estimated  additional  life-cycle  cost 
of  replicating  a  15  student  station  system  would  be  approximately  $54,000 
and  when  added  to  the  rest  of  the  pilot  system  increases  the  total  system 
cost  to  $158,000.  The  cost  of  providing  instruction  in  the  Airman 
Apprentice  course  is  estimated  at  $11.50  per  hour;  the  potential  annual  cost 
savings  which  could  be  realized  are  summarized  in  table  12.  The  course  cost 
estimate  was  derived  from  the  CNET  per  capita  course  costing  system. 


TABLE  12.  SUMMARY  OF  THE  POTENTIAL  ANNUAL  SAVINGS  MADE  POSSIBLE  BY  THE 
USE  OF  CAITS  AND  ATTAINED  BY  REDUCING  THE  LENGTH  OF  THE 
AIRMAN  APPRENTICE  COURSE 


Cost  Per 

Throughput 

Time  Reduction 

Cost  Reduction 

Per  Yr 

Course  Hour 

Per  Year 

Per  Student 

Per  Student 

Total 

$11.50 

2,000 

14.5  Hrs 

$166.75 

$333,500 

The  total  1 

ife-cycle  costs 

savings,  in  present 

value  terms. 

are  shown  i 

table  13. 


TABLE  13.  SUMMARY  OF  LIFE-CYCLE  SAVINGS  WHICH  ARE  POTENTIALLY 

POSSIBLE  WITH  THE  USE  OF  CAITS  IN  THE  AIRMAN  APPRENTICE 
COURSE  (In  Thousands) 


Year 


1  2  3  4  5  6  7  Total 


Annual  Savings  $334  $334  $334  $334  $334  $334  $334  $2,338 

Net  Present 

Value  2,155 

i 

Net  Present  ; 

Cost  15& 


$1,997 


Difference 
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The  additional  time  made  available  by  the  use  of  CAITS  could  be  used 
for  instruction  in  the  school  or  the  course  could,  in  fact,  be  shortened  by 
nearly  2  days  without  reducing  the  time  available  for  instruction.  An 
administrative  decision  would  be  necessary  to  capture  the  additional  time 
available.  It  is  doubtful  if  a  reduction  in  course  length  would  allow  all 
the  potential  savings  to  be  captured  in  the  short  run  but  in  the  long  run 
adjustments  in  the  management  and  structure  of  the  course  could  be  made  to 
realize  most  of  these  potential  savings.  The  magnitude  of  the  potential 
savings  is  such  that  there  can  be  little  doubt  that  the  use  of  CAITS  in  the 
Airman  Apprentice  course  leads  to  substantial  improvements  in  efficiency. 

One  might  question  whether  the  remediation  provided  by  the  computer  was 
as  effective  as  the  group  posttest  remediation.  The  findings  from  the 
previous  section  demonstrated  that  the  final  comprehensive  test  scores 
comparing  computer  remediated  students  and  conventional  posttest  remediated 
students  were  not  statistically  significant.  One  can,  therefore,  conclude 
that  the  performance  of  the  computer  remediated  student  was  no  lower  (or  no 
higher)  than  the  conventional  remediated  student  even  though  the  time  spent 
in  computer  remediation  was  less  than  10  percent  of  the  time  spent  in  the 
conventional  review  and  remediation  period.  This  time  reduction  certainly 
represents  an  improvement  in  training  efficiency.  No  analysis  was  conducted 
to  determine  if  either  form  of  remediation  significantly  changes  final 
comprehensive  test  scores. 

The  CAITS  provides  the  capability  to  record  student  responses  by 
content  area  and  thereby  provide  instructors  a  record  of  the  areas  where 
individual  student  performance  has  demonstrated  weaknesses.  The  CAITS  also 
provides  instructors  an  opportunity,  and  the  necessary  data,  to  focus  their 
remedial  efforts  on  the  specific  weakness  exhibited  by  students.  Summary 
results  of  student  performance  are  also  available,  and  these  provide 
information  on  course  content  areas  where  the  instructional  program  is  weak 
or  in  need  of  revision. 

Instructors  and  students  are  not  adverse  to  the  use  of  the  computer, 
and  there  are  recent  indications  that  students  prefer  computer  testing  after 
they  become  familiar  with  the  computer.  Users  of  CAITS  have  not  found  it 
difficult  to  use  in  the  operational  environment.  The  pilot  system  is 
presently  being  used  in  the  school  by  instructors  and  students  who  have  had 
limited  experience  with  this  type  of  computer  application. 

Other  anticipated  benefits  include  an  improvement  in  the  capability  to 
compile  and  report  administrative  data  on  students.  With  some  relatively 
minor  programming  effort,  the  individual  student  performance  data  could  be 
machine  entered  into  a  centralized  data  file  from  the  student  data  disks. 
Since  student  response  data  is  available  for  each  test  item,  the  summary 
data  from  the  CAITS  can  be  used  to  do  item  analysis  and  evaluate  reliability 
of  the  tests.  With  expanded  use  of  CAITS,  it  would  be  possible  to  establish 
a  centralized  test  analysis  capability  where  highly  skilled  personnel  could 
be  used  for  analysis  of  the  validity  and  reliability  of  tests. 
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SECTION  V 

CONCLUSIONS  AND  RECOMMENDATIONS 

This  section  contains  conclusions  regarding  the  effectiveness  and 
efficiency  of  the  CAITS  and  recommendations  for  the  use  of  CAITS  in  Airman 
Apprentice  and  other  Navy  schools.  The  conclusions  have  been  organized  into 
those  dealing  with  (1)  effectiveness  evaluation,  (2)  economic  evaluation, 
and  (3)  management  considerations. 

The  conclusions  involve  occasional  reference  to  the  treatment  groups 
employed  in  this  study.  These  groups  are  listed  again  to  facilitate  the 
presentation. 

•  Control  Group— traditional  paper  and  pencil  testing  with  1  hour 
allotted  for  testing  and  an  additional  hour  allotted  for  full 
class  review/remediation 

•  Experimental  Group  with  Computer  Remediation — Computer  based-test¬ 
ing  and  computer-based  remediation  with  remediation  being 

presented  only  to  students  who  have  not  satisfied  criteria  for 
specific  content  areas  within  test 

•  Experimental  Group  without  Computer  Remediation— computer-based 
testing  with  traditional  full  class  review/remediation. 

EFFECTIVENESS  EVALUATION  CONCLUSIONS 

1.  There  were  no  significant  differences  in  the  comprehensive  test 
scores  of  the  experimental  group  with  computer  remediation,  the  experimental 
group  without  computer  remediation  (traditional  review/remediation  only), 
and  the  control  group. 

2.  There  was  an  improvement  in  the  content  area  test  scores  for  the 
experimental  group  with  computer  remediation  immediately  following  remedia¬ 
tion.  Sixty-three  percent  of  the  remediations  resulted  in  a  passing  score  on 
the  post  remediation  test.  However,  the  relative  contributions  of 
remediation  and/or  retesting  could  not  be  directly  assessed  in  this  study. 
The  effect  of  various  forms  of  remediation  can  on’y  be  assessed  through  a 
controlled  experiment  designed  for  that  specif'  purpose.  There  was  no 
apparent  long-term  benefit  to  remediation  based  on  comprehensive  test 
content  area  scores. 

3.  Average  testing  time  for  computer-based  testing  was  slightly  less 
than  29  minutes  (both  experimental  groups)  plus  approximately  5  additional 
minutes  for  remediation  and  administrative  overhead  for  the  experimental 
group  with  computer  remediation.  The  additional  time  is  not  added  to  the 
experimental  group  without  computer  remediation.  For  noncomputer  (tradi¬ 
tional)  testing,  1  hour  is  allotted  for  testing  and  1  hour  for  review  and 
remediation. 
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ECONOMIC  EVALUATION  CON a US IONS 

1.  A  15  student  station  CAITS  used  as  a  testing  center  for  the  Airman 
Apprentice  course  at  Great  Lakes  has  the  potential  for  saving  14,500  student 
hours  per  year.  Translating  student  hour  savings  into  real  cost  savings  or 
improvements  in  effectiveness  will  require  that  management  reduce  the  length 
of  the  Airman  Apprentice  course  or  effectively  utilize  the  saved  student 
time. 


?.  The  present  value  cost  for  a  7-year  life  cycle  of  the  first  CAITS 
was  $94,022.  The  present  value  cost  for  a  7-year  life  cycle  of  a  CATTS 
replication  would  be  $54,097. 

3.  The  CAITS  offers  nonquantif iable  benefits  in  areas  such  as 
immediate  student  performance  feedback,  class  level  data  collection  which 
can  be  machine  transmitted  to  various  data  bases  for  administrati ve  or 
management  purposes,  capabilities  for  test  item  analysis,  and  individualized 
remed i at i on . 

4.  Based  on  cost  per  course  hour  and  time  reduction  realized  with 
CAITS,  a  long  run  potential  exists  for  savings  worth  approximately  2  million 
dollars  during  a  7-year  system  life  cycle  for  Airman  Apprentice  training  at 
Great  Lakes.  These  potential  savings  are  contingent  upon  the  continued 
operation  of  the  present  CAITS,  adding  an  additional  15  student  station 
CAITS  testing  center,  and  managing  the  Airman  Apprentice  course  to 
effectively  utilize  the  saved  time  or  reduce  the  course  length. 

MANAGEMENT  CONSIDERATIONS 

1.  The  CAITS  project  demonstrated  the  feasibility  of  using  Program 
Development  Center  personnel  for  high  technology  development  when  personnel 
with  specialized  skills  from  other  NAVEDTRACOM  activities  are  available  for 
support. 


2.  The  collocation  of  the  development  activity  with  the  using 
activity  facilitated  development  and  implementation  of  CAITS.  This  gave  the 
usinq  activity  a  sense  of  partnership  in  development  which  resulted  in  an 
efficient  transition  from  development  to  school  implementation. 

3.  Computer-based  remediation  serves  to  correct  test  performance 
deficiencies  in  two-thirds  of  the  remediation  cases  for  the  short  term.  The 
remaining  one-third  of  the  remediation  cases,  which  represents  approximately 
three  percent  of  all  content  area  testing,  must  be  instructor  remediated. 
This  indicates  that  either  additional  computer  remediation  may  be  required 
or  that  allowance  should  be  made  for  the  inclusion  of  instructor  time  to 
provide  this  remediation. 

RECOMMENDATIONS 

1.  Expand  the  existing  CAITS  at  Airman  Apprentice  School,  Great 
Lakes,  by  adding  a  second  15  station  testing  center.  This  will  provide  the 
capacity  to  accomplish  all  Airman  Apprentice  testing  by  CAITS  at  Great  Lakes 
and  realize  the  projected  savings  discussed  in  section  IV. 
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2.  Evaluate  the  economic  feasibility  of  implementing  the  CAITS  for 
all  apprentice  testing  at  the  three  apprentice  training  locations. 

3.  Evaluate  the  feasibility  of  utilizing  the  CAITS  concept  for 
recruit  testing  at  the  three  Recruit  Training  Commands. 
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SOFTWARE  DEVELOPMENT  ASSESSMENT 
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TAEG:MMZ 

19  February  1981 


From:  Director,  Training  Analysis  and  Evaluation  Group 
To:  Chief  of  Naval  Education  and  Training  (N-932) 

Subj:  Alternative  means  of  Computer-Assisted/Interactive  Testing  (CAIT) 
Testing  Strategy  and  Software  Development 

Ref:  (a)  CNET  Itr  Code  022  of  6  Jan  1981 

(b)  NAVEDTRAPRODEVCENDET,  Great  Lakes  Proposal,  "Computer  Assisted/ 
Interactive  Testing  (CAIT)  Strategy  Proposal  for  Airman 
Apprentice  4.0  Week  Expansion  Program" 

1.  Reference  (a)  reguested  the  TAEG  to  review  reference  (b)  and  examine 
alternative  means  of  accomplishing  the  development  of  the  testing  strategy 
and  software  program.  Findings,  including  statements  of  total  time  to 
completion  and  all  costs,  were  to  be  submitted  to  CNET  along  with 
recommendati ons . 

2.  A  meeting  was  held  between  members  of  the  TAEG  and  NAVEDTRAPRODEVCENDET, 
Great  Lakes,  on  5  and  6  February  1981  to  outline  details  of  the  testing 
strategy  and  to  examine  the  capabilities  of  present  eguipment  with  regard  to 
the  implementation  of  the  testing  system. 

3.  Agreement  was  reached  between  TAEG  and  NAVEDTRAPRODEVCENDET,  Great 
Lakes,  on  the  minimum  functional  specifications  to  be  met  by  the  CAIT, 
including  testing  strategy,  remediation  strategy,  management  information 
reguirements,  and  design  constraints. 

4.  It  was  concluded  that  the  CAIT  proposal  in  reference  (b),  as  modified 
during  the  discussions  of  5  and  6  February,  is  not  only  feasible  but  also 
has  potential  for  significant  training  efficiencies.  However,  the  current 
hardware  configuration  cannot  meet  system  design  specifications.  Additional 
equipment  is  required  to  fully  implement  the  system. 

5.  It  is  recommended  that  a  32-station  testing  system  be  implemented  with 
dual  16-station  subsystems.  This  will  insure  adequate  system  reliability 
and  backup  capability.  The  additional  hardware  components  necessary  to 
complete  the  test.inq  configuration  will  require  $24K  of  0&MN  funding.  These 
equipment  costs  remain  the  same  reqardless  of  the  alternative  chosen  for 
software  development. 

6.  Three  alternatives  have  been  identified  for  the  development  of  the 
system.  These  are  identified  below  alonq  with  their  associated  costs  and 
development,  time. 
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TAEG:MMZ 

19  February  1981 

Subj:  Alternative  means  of  Computer-Assisted/Interactive  Testing  (CAIT) 
Testing  Strategy  and  Software  Development 


HARDWARE 

SOFTWARE 

CALENDAR  MONTHS 

ALTERNATIVE 

DEVELOPMENT 

DEVELOPMENT 

TO. 

COMPLETION 

1 

IN  HOUSE 

( $24K  EQUIPMENT) 

CONTRACT 

(DESIGN  &  PROGRAMMING  $50K) 

12 

2 

IN  HOUSE 

( $24K  EOUIPMENT) 

IN  HOUSE 

(DESIGN  &  PROGRAMMING  BY 

IN  HOUSE  PERSONNEL) 

18 

3 

IN  HOUSE 

(S24K  EQUIPMENT) 

IN  HOUSE 

(DESIGN  BY  IN  HOUSE  PERSONNEL 
CONTRACT  PROGRAMMING  $18K) 

8 

In  addition,  there  may  be  a  requirement  for  operating  system  modification  by 
the  equipment  vendor  ($10K). 

7.  It  is  recommended  that  alternative  3  be  implemented.  This  alternative  has 
a  relatively  short  development  time  and  minimal  costs.  In  addition,  it 
maintains  the  testing  and  system  expertise  in  house.  This  expertise  would  be 
particularly  valuable  should  a  decision  be  made  to  expand  the  system  to  other 
sites.  Although  alternative  2  has  the  same  desirable  features  of  alternative 
3,  it  is  not  recommended  because  the  use  of  in  house  personnel  will  have 
negative  impacts  on  the  completion  of  other  programs  at  NAVEDTRAPRODEVCENDET. 
Alternative  1  is  not  recommended  because  of  its  relatively  high  cost  and 
because  all  system  software  expertise  would  reside  with  the  contractor. 

8.  For  all  alternatives  TAEG  will: 

a.  Provide  functional  performance  specifications 

b.  Provide  hardware/software  system  design  specifications 

c.  Conduct  progress  reviews  and  an  operational  test  and  evaluation  of  the 
complete  system 

d.  Conduct  an  economic  evaluation  of  the  operational  test  system. 


/s/ 

A.  F.  SMODE 

Copy  to: 

CNET  (Code  02,  022) 

DIR  NAVEDTRAPRODEVCENDET,  GLAKES 
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FUNCTIONAL  SPECIFICATIONS 
COMPUTER  AIDED  INTERACTIVE  TESTING  SYSTEM 
GREAT  LAKES,  ILLINOIS 

The  functional  performance  characteristics  described  in  this  specification 
are  to  be  incorporated  into  the  design  of  the  Computer  Aided  Interactive  Test¬ 
ing  System  (CAITS). 

ADMINISTRATIVE  SUBSYSTEM 

1.  Registration 

a.  Instructors  will  enroll  students  into  the  CAITS.  As  a  minimum, 
enrollment  information  will  include  SSN,  name,  class,  and  other  biographical/ 
performance  data  normally  maintained  on  student  "hard  cards." 

b.  The  CAITS  student  record  will  provide  for  entries  concerning: 

(1)  Setback 

(2)  Attrition 

c.  Only  instructor  and  staff  personnel  will  be  authorized  to  change 
student  record  data. 

d.  The  administrative  subsystem  will  provide  hard  copy  output  to  be 
used  as  class  rosters  for  instruction. 

e.  Information  contained  in  the  administrative  subsystem  will  be 
maintained  in  standard  ASCII  format  to  make  it  suitable  for  transfer  to  other 
NAVEDTRACOM  management  information  systems. 

2.  Progress  Reports 

a.  At  the  completion  of  each  test,  the  system  will  provide  a  student 
status  report  containing  three  parts: 

(1)  (a)  Alpha  and  SSN  students  who  passed/w.  score/subpart  score, 
(b)  Alpha  and  SSN  students  who  failed/w.  score/subpart  score. 

(2)  Alpha  and  SSN  students  who  missed  test  for  authorized/ 
unauthorized  reasons. 

(3)  Alpha  and  SSN  students  assigned  off  line  remediation  and 
assignments  (content  areas  requiring  remediation). 

3.  The  administrative  subsystem  will  provide,  on  demand,  a  report  contain¬ 
ing  the  alpha  and  SSN  of  students  dropped  in  a  particular  time  frame.  It 
will  also  include  the  appropriate  codes  for  separation  reasons  as  applicable 
in  the  school.  These  data  will  be  maintained  on  a  class,  quarterly,  and/or 
annual  basis. 
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4.  The  administrative  subsystem,  on  demand,  will  provide  a  setback/advance 
report.  This  report  will  contain  the  number  of  students  advanced/setback  in 
a  particular  time  frame.  It  will  also  provide  the  appropriate  codes  used  by 
the  school  to  report  these  actions  to  higher  authority.  These  data  will  be 
maintained  on  a  class,  quarterly,  and/or  annual  basis. 

5.  Student  Performance  Summary.  The  administrative  system  will  report  the 
performance  score  of  individual  students  at  any  given  time.  This  report  will 
include  grades,  exams  missed,  and  number  of  remediations  by  content  area,  and 
for  the  duration  of  system  development  and  evaluation  individual  student 

•records  will  be  maintained  in  computer  compatible  archival  form  at  IPDC  DET, 
Great  Lakes.  It  is  recommended  that  consideration  of  long  term  storage  of 
archival  data  be  included  in  the  Instructional  Management  Plan  (IMP).  Records 
will  be  maintained  in  the  archives  for  2  years  to  insure  availability  for 
longitudinal  research  and/or  evaluation  projects. 

TESTING  SUBSYSTEM 

1.  Testing  Strategy 

a.  A  test  will  be  organized  so  as  to  evaluate  each  content  area  in  an 
instructional  unit.  Test  items  will  be  keyed  to  specific  content  areas  and 
learning  objectives.  The  number  of  test  items  will  vary  with  the  number  of 
learning  objectives  being  tested.  Generally,  the  maximum  number  of  items 
shall  be  50  and  the  minimum  10. 

b.  Test  performance  of  each  student  will  be  immediately  evaluated  on¬ 
line  against  performance  criteria  established  by  the  curriculum  developer. 

This  assessment  will  consist  of  evaluation  of  both  individual  item  responses 
and  patterns  within  content  areas.  For  those  items  or  content  areas  on  which 
students  fail  to  reach  criterion  levels,  the  testing  strategy  requires  that 
remediation  of  either  a  specific  or  general  nature  be  provided,  based  on 
remediation  requirements  established  by  the  instructional  developer. 

c.  Immediately  following  on-line  remediation,  students  will  be  adminis¬ 
tered  a  second  form  of  the  test.  This  test  is  to  be  composed  of  a  second 
form  of  all  content  areas  previously  remediated  to  include  all  critical 
items,  as  defined  by  the  instructional  developer. 

d.  Failure  on  a  test  following  remediation  (Test  II)  will  require  the 
student  to  be  assigned  remediation  offline  and/or  to  be  referred  to  the 
instructor  fur  further  guidance.  The  instructor  will  be  required  to  verify 
off-line  remediation  prior  to  administering  a  third  test  covering  all  content 
areas  originally  tested.  Failure  on  this  final  examination  in  a  topic  area 
will  require  action  in  accordance  with  school  policies  in  effect  at  that 
time. 

2.  Test  Construction  and  Analysis 

a.  A  pool  of  test  items  sufficient  in  size  to  permit  construction  of 
three  alternative  and  parallel  forms  of  each  test.  Items  will  employ  a 
multiple  choice  response  format.  Question  stems  may  be  of  any  form  which 
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permits  objective  responses.  Centralized  storage  for  a  maximum  of  3,000 
items  will  be  provided.  Student  stations  will  store  one  complete  test  with 
remediations. 

b.  All  items  will  be  keyed  to  specific  learning  objectives  and  content 
areas.  Tests  of  an  appropriate  size  for  each  of  the  instructional  units  will 
be  constructed  by  the  computer. 

c.  Item  distractors  are  to  be  either  partially  or  totally  randomized 

for  delivery  and  coded  in  such  manner  that  all  student  responses  can  be  returned 
to  the  original  test  format  prior  to  randomization  for  delivery.  Unless 
otherwise  specified  the  default  condition  will  be  total  randomization  of 
responses. 

d.  During  the  development,  test,  and  implementation  of  the  CAITS,  test 
authoring  and  analysis  capability  will  reside  at  NETPDCDET,  Great  Lakes. 
Responsibility  for  test  development  and  analysis  may  be  transferred  to  the 
school  subsequent  to  system  development. 

e.  All  tests  (including  parallel  forms)  will  be  maintained  on  a  central 
computer  fixed  disk.  This  storage  will  include  at  maximum  a  total  of  10 
unique  tests  varying  in  length  from  10  to  50  items  (each  having  additional 
parallel  forms)  and  associated  remediation  for  each  test. 

3.  Data  Storage  and  Analysis 

a.  Each  test  to  be  administered  shall  be  duplicated  in  its  entirety 
from  the  central  computer  file  onto  a  read  only  floppy  disk.  Each  student  is 
to  receive  a  disk  containing  the  appropriate  test,  associated  remediation 
material,  and  a  second  form  of  the  test.  Each  student  terminal  is  to  act  in 
stand  alone  mode  for  each  test.  Software  located  in  the  student  terminal 

will  administer  and  evaluate  test  performance  and  assign  necessary  remediation. 
Following  initial  remediation,  this  stand  alone  system  will  administer  the 
required  second  test  materials.  Students  failing  the  second  test  will  be 
assigned  either  to  off-line  remediation  and/or  to  the  instructor  for  further 
guidance. 

b.  The  entire  test  battery  shall  be  maintained  under  secure  conditions 
at  the  school  house  in  accordance  with  Test  Control  Officer  policies. 

c.  Tests  shall  be  transferred  from  fixed  disk  to  individual  floppy 
disk  for  distribution  to  each  student.  A  complete  inventory  of  all  tests  on 
floppy  disks  will  be  maintained  under  secure  conditions  at  the  schoolhouse. 
Floppy  disks  available  to  student  shall  be  "read  only." 

d.  All  student  response  data  will  be  stored  on  floppy  disk  memory  at 
the  student  station  until  such  time  as  the  student  has  completed  the  testing 
session;  at  that  time  all  response  data  will  be  transferred  to  the  student's 
permanent  record  in  the  control  computer. 

e.  Student  data  to  be  collected  and  transferred  to  the  central  computer 
shall  include: 
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(1)  distractor  selected  on  each  item  by  each  student 

(2)  correct  or  incorrect  grade  or  response  on  each  item 

(3)  time  required  to  complete  item 

(4)  number  of  items  and  content  areas  remediated  per  test 

(5)  number  of  remediations  per  content  area  and  per  test 

(6)  final  grade  on  test 

(7)  pass/fail 

(8)  setback/attrite 

(9)  test  form  identifying  information. 

f.  The  system  will  also  report  the  student  start  and  stop  times  on  the 
test  and  the  date.  These  will  be  recorded  by  module,  content  area,  or  entire 
course  and  will  be  summarized  on  an  individual  student  and  class  basis.  Times 
taken  for  individual  items  by  students  will  be  monitored  and  recorded 

and  when  a  criterion  level  established  by  curriculum  developers  has  been 
reached,  the  system  will  set  a  warning  flag.  The  flag  will  be  indicated  at 
the  instructor  station  console  and  be  used  to  monitor  potential  performance 
problems. 

g.  The  system  will  limit  each  multiple  choice  item  and/or  graphics 
item  to  one  page  (video  display  screen). 

4.  NETPDCDET,  Great  Lakes,  will  have  initial  responsibility  of  the  maintenance 
of  an  overall  and  complete  response  history  of  each  item  in  the  curriculum. 
Subsequent  to  the  development  of  test  and  implementation  of  the  CAITS,  this 
responsibility  will  be  assumed  by  the  school.  Response  histories  will  be  updated 
periodically;  e.g.,  monthly,  consistent  with  the  overall  system  design  for 
maintenance  of  response  information.  These  items  will  be  retained  in  an  item 
bank  (archive)  and  periodically  reexamined  for  reliability  and  validity.  The 
item  bank  will  be  so  constructed  as  to  maintain  items  by  content  area  and 
learning  objectives. 

5.  An  off-line  item  analysis  capability  will  be  maintained  at  NETPDCDET. 

These  analyses  will  consist  of  item  difficulty,  item  discriminabil ity,  reliability, 
and  validity  (if  data  are  available).  Periodic  revision  of  tests  shall  be 
undertaken  based  on  these  analyses.  These  evaluations  shall  be  undertaken  at 
any  time  an  item  analysis  suggests  a  revision  may  be  required,  but  at  a 
minimum,  revisions  shall  be  undertaken  every  2  years.  Item  analyses  by  class 
shall  be  made  available  on  request. 

6.  Periodically;  e.g.,  monthly  or  quarterly,  response  information  will  be 
stripped  from  the  files  of  the  central  computer  and  be  transmitted  to  the 
Evaluation  Branch  of  the  NETPDCDET.  These  data  will  be  merged  with  those 
resident  in  the  item  data  bank. 

TEST  DELIVERY  SYSTEM 

1.  The  system  shall  be  equipped  to  accommodate  an  average  class  size  of  32 
students  at  one  sitting.  Each  student  testing  station  is  to  be  provided  a 
microcomputer  with  video  display. 
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2.  All  tests  will  also  be  maintained  in  paper  and  pencil  format  to  serve  as 
backup  should  the  primary  computer  based  system  fail.  These  paper  versions 
of  the  test  are  to  be  maintained  in  a  secured  location  under  control  of  the 
school.  Any  revisions  to  the  computer  based  test  will  be  reflected  in  the  manual 
testing  system  as  necessary. 

3.  The  system  will  provide  the  capability  for  on-line  monitoring  of  student 
station  progress  at  the  option  of  the  instructor.  This  capability  is  not  to  act 
to  simply  repeat  current  information  on  a  student  station  display  but  is  to 
permit  a  selective  search  of  student  test  record. 

4.  All  files  in  the  central  computer  shall  employ  a  file  protect  feature  to 
guard  against  accidental  erasure  or  overwrite;  e.g.,  power  outages.  A  con¬ 
tingency  plan  will  be  specified  prior  to  system  installation  for  booting  any 
backup  files. 

5.  The  test  delivery  system  shall  be  designed  to  allow  for  future  expansion 
of  capabilities  in  CAI  and  testing  and  to  permit  fully  automatic  interactive 
testing  under  the  control  of  a  central  computer  without  intervening  manual 
transfer  of  information  to  individual  test  sites.  The  system  will  also  allow 
for  the  eventuality  of  exporting  instruction/testing  to  remote  sites. 

Student  stations  should  be  designed  to  permit  stand-alone  operation  and/or 
permit  integration  into  local  area  instructional  networks. 
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1.0  SCOPE 

This  specification  covers  the  hardware  and  software  requirements  for  the 
design,  development,  and  test  of  the  Computer  Aided  Interactive  Testing  System 
(CAITS).  The  CAIT  System  will  conform  to  functional  specifications,  No  1170-FS 
of  June  1981,  and  will  satisfy  training  requirements  developed  under  NAVEDTRA 
106A  and  NAVEDTRA  110.  In  the  event  of  conflict  between  CAITS  requirements  and 
training  system  requirements,  training  system  requirements  will  take  precedence. 

The  CAIT  System  specified  herein  is  intended  to  satisfy  a  portion  of  the 
testing  requirement  for  the  Airman  Apprentice  Training  Program  at  RTC,  Great 
Lakes.  Additionally,  it  will  have  general  applicability  to  other  Navy  training 
programs.  It  will  function  as  an  integrated  testing  system  as  described  in  this 
specification  for  group  testing  application  and  will  also  function  as  a  single 
station,  stand-alone  system  with  a  combined  text  and  computer  graphic  capability. 
These  features  allow  a  range  of  computer  based  instructional  applications 
including  instructional  delivery  and  instructional  management.  This  enhanced 
capability,  if  desired,  can  be  provided  by  developing  additional  applications 
software.  It  will  also  be  possible  to  expand  data  storage  and  to  provide 
special  purpose  input-output  capabilities  by  adding  plug  compatible  hardware. 

With  this  flexibility  it  is  envisioned  that  other  training  requirements  can  be 
satisfied  with  incremental  additions  of  software  and  hardware. 

2.0  SYSTEM  REQUIREMENTS 

The  CAITS  will  utilize  commerically  available  computer  hardware  subsystem 
components  wherever  possible.  Processors,  disk  storage  units,  printers,  display/ 
keyboard  units,  controllers  and  interface  logic  should  be  selected  on  the  basis 
of  performance,  intra-system  compatibility,  supportabil i ty  and  cost.  In  addition, 
the  hardware  portion  of  the  CAITS  will  support  necessary  system  features  such  as 
system  diagnostics  and  general  system  utilities. 

The  development  software  for  the  CAITS  will  consist  of  vendor  supplied 
operating  systems,  interpreters,  compilers,  assemblers,  editors,  file  management 
systems  and  general  utilities.  The  operational  software  will  consist  of  the 
applications  programs  for  the  instructor  station  and  the  student  stations  together 
with  the  operating  system,  interpreter  and  monitor  software/ firmware.  The 
hardware,  software,  and  testing  requirements  for  the  CAIT  System  are  considered 
to  be  deliverables  and  are  addressed  in  paragraphs  2.1,  2.2,  and  3.0  of  this 
specification. 

2.1  HARDWARE  REQUIREMENTS 

The  hardware  conf iguration  for  the  CAIT  System  is  shown  in  figure  (1).  As 
shown,  the  system  consists  of  a  two  Microcomputer  Instructor  Station,  16  single 
microcomputer  student  stations  and  interface  logic.  Two  16  station  systems  as 
shown  in  figure  (1)  will  constitute  a  32  station  testing  system. 

2.1.1  INSTRUCTOR  STATION 

An  Instructor  Station  will  consist  of  an  Ohio  Scientific  C3-D  microcomputer 
system  with  parallel  interface  to  a  Bell  and  Howell  3048D  microcomputer  system. 
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Each  of  these  computers  operate  under  vendor  supplied  operating  systems  and 
special  purpose  applications  software.  Program  compatibility  between  computers 
will  not  be  required  since  each  processor  will  provide  dedicated  computer 
functions.  Data  interchange  will  be  in  accordance  with  predefined  data  formats 
and  communications  protocol.  Communications  control  will  follow  a  master-slave 
philosophy. 

2. 1.1.1  INSTRUCTOR  STATION  DISK  STORAGE 

The  Ohio  Scientific  C3-D  Microcomputer  System  includes  an  integral  10 
megabyte  hard  disk  unit  and  an  8  inch  floppy  disk  unit.  The  hard  disk  will 
contain  all  of  the  applications  programs,  student  records,  test  masters  and 
control  Programs.  The  floppy  disk  will  be  used  for  program  transport,  hard  disk 
back-up  and  archival  storage. 

The  Instructor  Station  Bell  and  Howell  dual  floppy  disk  unit  will  be  used  to 
generate  student  station  floppies  for  student  testing.  The  test  item  selection 
and  sequencing  will  be  under  test  author  control  and  will  be  in  accordance  with 
the  CAIT  System  functional  specification. 

2. 1.1. 2  KEYBOARD  DISPLAY 

The  instructor  station  will  include  two  keyboard-display  units,  one  for  the 
Ohio  Scientific  microcomputer  and  one  for  the  Bell  and  Howell  microcomputer. 
Although  a  single  keyboard  display  could  be  utilized  for  both  microcomputers, 
only  one  computer  at  a  time  would  be  operational.  Although  this  would  not 
normally  affect  system  operation,  there  will  be  occasions  when  it  is  desirable 
to  make  floppy  disk  copies  at  the  Bell  and  Howell  station  while  test  management 
is  taking  place  at  the  Ohio  Scientific  station. 

2. 1.1. 3  PRINTER 

A  printer  is  required  at  the  Instructor  Station  to  provide  hard  copy  output 
in  many  functional  areas.  Some  specific  examples  are  student  progress  reports, 
class  testing  schedules,  evaluation  reports,  test  masters,  system  diagnostic 
records  and  test  center  management  information  reports.  Although  the  printer 
will  normally  be  used  as  an  Ohio  scientific  microprocessor  peripheral,  it  should 
be  capable  of  functioning  as  a  Bell  and  Howell  microprocessor  peripheral  without 
special  adaptation.  Because  this  printer  will  be  used  in  the  testing  center, 
selection  factors  should  include  low  noise,  high  reliability,  medium  print 
speed,  good  print  quality  and  graphics  reproduction  capability.  A  high  quality, 
high  dot  density  matrix  impact  printer  appears  to  be  a  reasonable  choice  but 
thermal,  ink  jet  and  daisy  wheel  options  should  be  considered  in  the  selection 
process. 

2.1.2  STUDENT  STATION 

Each  student  station  will  consist  of  a  Bell  and  Howell  3048D  microcomputer, 
a  monitor  for  display  and  a  Bell  and  Howell  dual  floppy  disk  drive.  The  keyboard 
portion  of  the  keyboard-display  capability  is  an  integral  part  of  the  micro¬ 
computer  system  and  will  function  as  the  student  test  response  input  device. 
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Each  student  station  microcomputer  conmini cates  with  the  host  Ohio  Scientific 
computer  via  a  serial  RS-232C  line.  This  communications  link  provides  for 
information  interchange  relating  to  student  identification,  test  item  response, 
student  station  status  and  test  management/control.  All  testing  data  (test 
items,  remediations  and  graphics)  will  be  contained  on  floppy  disk.  Distribution 
control  for  the  test  floppies  will  be  maintained  by  the  instructor/proctor. 

2. 1.2.1  STUDENT  STATION  DISK  STORAGE 

The  Bell  and  Howell  dual  floppy  disk  drives  at  each  student  station  will 
be  utilized  during  both  integrated  system  operation  and  stand-alone  operation. 

The  number  one  drive  will  contain  the  system  disk  and  will  provide  temporary 
storage  for  test  item  responses.  During  integrated  system  operation  these 
responses  will  be  transmitted  to  the  central  instructor  station  as  required. 
During  stand-alone  operation  the  responses  will  be  stored  on  floppy  disk  until 
action  is  taken  by  proctor  to  transport  the  disc  to  a  data  reduction  location. 

The  number  two  drive  will  contain  the  test  floppy.  This  floppy  will  be  write 
protected  and  software  read  protected  to  maintain  test  security.  The  data 
format  on  both  floppies  will  be  in  accordance  with  Bell  and  Howell  specifications 
and  will  not  be  compatible  with  the  Ohio  Scientific  floppy  format  or  the  Ohio 
Scientific  disk  operating  system.  It  will,  however,  be  fully  compatible  with 
the  instructor  station  Bell  and  Howell  microcomputer  disk  operating  system. 
Computer  to  computer  data  interchange  between  Ohio  Scientific  and  Bell  and 
Howell  will  provide  the  necessary  link  to  interchange  disk  data  controlled  by 
different  disk  operating  systems. 

2.1.3  INTERFACE  AND  COMMUNICATIONS 

The  hardware  interface  between  microprocessor  and  standard  peripherals 
(keyboard-display,  printer,  disk  drive)  will  be  standard  vendor  items  for  both 
the  Ohio  Scientific  and  Bell  and  Howell  microprocessors.  The  driver  software 
will  also  be  standard  for  each  vendor's  software  package  with  the  possible 
exception  of  motherboard  slot  placement  coding.  This  standard  hardware-standard 
software  approach  to  system  design  and  development  should  be  followed  wherever 
possible  in  the  interest  of  reducing  system  design  complexity.  It  will  also 
decrease  risk  and  increase  the  probability  of  meeting  design  and  development 
schedules. 

The  standard  equipment  approach  cannot  be  followed  fully  in  the  case  of 
interface  between  microprocessors  of  different  manufacturers  such  as  the  Ohio 
Scientific  and  Bell  and  Howell  in  this  system.  For  this  case  (instructor 
station  to  student  station  transfer)  the  serial  interface  will  be  standard  RS- 
232C  at  the  highest  baud  rate  possible  within  the  design  limits  of  the  vendor 
supplied  hardware  and  software.  Standard  serial  interface  boards  with  minor 
modification  should  allow  for  4800  baud.  This  option  should  be  investigated 
during  the  design  phase.  Serial  interface  will  be  the  standard  communications 
link  between  instructor  station  and  student  stations.  It  will  also  allow  for 
future  modem  linking  for  remote  operation. 

Although  the  hardware  interface  and  communications  protocol  are  fully 
defined  for  the  computers  to  be  used,  the  software  system  is  not  defined. 
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Information  transfer  can  take  place  in  many  forms  (Text,  Basic  Language  State¬ 
ments,  Binary,  Block  Data  Structure)  and  the  form  of  data  formatting  for  data 
interchange  should  be  considered  carefully  during  the  design  phase.  This 
information  transfer  consideration  must  also  include  computer  graphic  informa¬ 
tion  which  is  handled  and  formatted  differently  for  each  vendors  microprocessor. 
Binary  block  (packet)  transfer  is  suggested  for  this  application. 

The  two  microcomputers  at  the  instructor  station  are  also  of  different 
manufacture  and  must  communicate  at  high  data  rates  to  provide  flexibility  in 
utilizing  hard  disk  storage  from  the  Bell  and  Howell  work  station.  A  standard 
parallel  interface  should  be  utilized  for  this  computer  to  computer  application. 
Special  software  will  be  required  to  provide  the  necessary  data  handling  between 
computers.  Terminal  emulation  is  the  preferred  design  approach  since  operating 
system,  monitor  and  driver  modification  should  not  be  considered.  Assembly 
language  subroutines  will  provide  special  features  where  required. 

2.2  SOFTWARE  REQUIREMENTS 

The  software  development  effort  for  the  CAITS  will  consist  of  designing 
and  developing  applications  programs  to  satisfy  the  Functional /Performance 
Specification  Requirements.  The  documentation  requirements  for  the  CAIT  System 
will  be  limited  to  the  Program  Performance  Specification,  Interface  Design 
Specification,  Program  Design  Specification,  Data  Base  Design  Document,  Program 
Description  Document,  and  Users  Manual.  The  System  Test  Procedure  and  Report 
will  be  addressed  separately  in  Section  3.0,  Testing  Requirements.  A  top  down 
design  approach  will  be  followed  which  will  define  programs  and  program  modules 
on  the  basis  of  functional  performance  relating  to  the  system  specification.  The 
top  down  structure  shall  support  functional  modularity  for  ease  of  coding, 
system  integration  and  subsequent  program  changes. 

The  functional  requirements  for  the  CAIT  System  are  delineated  in  the  CAITS 
Functional  Specification  No.  1170-FS  of  1  June  1981.  This  specification  constitutes 
the  functional  baseline  from  which  the  Program  Performance  Specification  will 
be  developed.  The  Program  Performance  Specification,  when  approved,  will 
become  the  master  design  and  configuration  control  document  for  CAITS  program 
development. 

2.2.1  PROGRAM  PERFORMANCE  SPECIFICATION 

The  Program  Performance  Specification  shall  describe  the  operational  and 
functional  requirements  necessary  to  design,  test  and  maintain  the  CAITS 
programs.  It  will  provide  detailed  descriptions  of  program  performance  and 
the  relationship  of  program  functions  to  system  functional  performance  require¬ 
ments.  The  Program  Performance  Specification  (PPS)  will  be  used  as  a  controlling 
document  for  the  design  and  test  of  programs  developed  for  the  CAIT  System. 

Data  Item  Description,  DI-E-21 36/MOD,  Specification,  Program  Performance  (PPS) 
of  2  July  1979  will  be  used  for  guidance  in  the  development  of  the  CAITS  PPS. 

2.2.2  INTERFACE  DESIGN  SPECIFICATION 

The  Interface  Design  Specification  (IDS)  will  define  the  inter/intra 
processor  interfaces  in  the  CAIT  System.  It  should  provide  a  description  of  the 
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hardware  and  software  interface  logic,  communications  conventions,  data  types/ 
formats  and  data  communication  rates.  Interface  Design  Specification  (IDS)  Data 
Item  Description  DI-E-21 35/MOD  will  be  used  for  general  guidance  only.  The 
CAITS  IDS  will  provide  fully  descriptive  information  for  all  CAITS  Interface 
(program  and  physical).  This  document  must  be  maintained  under  configuration 
control  since  it  will  be  a  key  document  during  system  test  and  follow-on  life 
cycle  support. 

2.2.3  PROGRAM  DESIGN  SPECIFICATION 

The  Program  Design  Specification  (PDS)  will  follow  the  guidelines  established 
in  Data  Item  Description  DI-E-21 38/MOD.  A  work  breakdown  structure  will  not  be 
required  for  the  CAIT  System.  However,  functional  allocation  should  be  identifi¬ 
able  and  traceable  from  the  functional  performance  specification  to  programming 
module  flow  chart/di agram/ model .  The  program  Design  Specification  will  be  the 
software  design  document  for  each  program  module  and  as  such  should  provide  the 
programmer  with  fully  descriptive  information  for  program  coding. 

2.2.4  DATA  BASE  DESIGN  DOCUMENT 

The  Data  Base  Design  Document  (DBDD)  will  provide  a  complete  and  detailed 
description  of  all  common  data  items  (variables,  strings,  arrays,  constants, 
booleans)  to  be  used  by  more  than  one  program  module.  This  document  will  be  of 
great  importance  in  the  design  philosophy  to  be  followed  when  using  the  Basic 
Interpretive  Language.  Modularization  of  programs  will  require  data  initialization 
and  data  reset  inhibit  features  beyond  those  normally  available  with  the  inter¬ 
preter  firmware.  Special  programming  provisions  will  be  necessary  for  handling 
common  data  items.  Data  Item  Description  DI-S-214Q/M0D  should  be  followed 
during  the  preparation  of  this  document. 

2.2.5  PROGRAM  DESCRIPTION  DOCUMENT 

The  Program  Description  Document  (PDD)  shall  provide  complete  technical 
descriptions  of  CAIT  System  module  functions,  structures,  operational  constraints, 
source  and  object  listings,  diagrammatic/narrative  flows  and  data  base  organiza¬ 
tion.  Data  Item  Description  DI-S-21 39/MOD  will  be  used  as  a  guideline  during 
the  development  of  this  document. 

2.2.6  USERS  MANUAL 

The  Users  Manual  will  provide  an  operational  description  of  the  CAIT  System. 

It  will  also  provide  detailed  procedures  for  all  operating  modes  (test  delivery, 
test  authoring,  report  generation,  evaluation).  The  procedural  information 
will  be  clearly  presented  in  user  (non- programmer)  language  and  will  be 
modularly  formatted.  This  manual  should  contain  all  of  the  information  required 
for  system  operation  without  referring  to  other  documentation  since  it  will  be 
the  only  manual  available  during  normal  operation. 

3.0  TESTING  REQUIREMENTS 

All  hardware,  software,  and  system  tests  will  be  conducted  in  accordance 
with  approved  test  procedures.  Hardware  testing  will  consist  of  diagnostic 
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testing,  mechanical  inspection  and  limited  reliability  testing.  Software  testing 
will  consist  of  module  testing  and  integration  testing.  System  tests  will 
demonstrate  that  functional  performance  requirements  as  delineated  in  the  CAITS 
functional  performance  specification  have  been  satisfied. 

3.1  SYSTEM  TEST  PROCEDURE 

The  System  Test  Procedure  for  the  CAITS  will  ensure  that: 

a.  The  system  is  tested  in  the  operational  environment  under  high  stress 
conditions 

b.  All  operational  modes  are  tested 

c.  Recovery  is  possible  without  reinitialization  for  user  generated 
faults 

d.  Student  testing  time  is  not  constrained  by  computer 

e.  Data  communications  standards  are  satisfied 

f.  Modular  testing  of  software  is  possible 

g.  All  System  Performance  Specification  requirements  are  satisfied 

h.  All  System  Design  Specification  requirements  are  satisfied 

i.  Documentation  reflects  the  system  design  and  configuration. 

The  test  procedures  will  be  prepared  by  the  design  and  development  group 
and  will  be  submitted  for  approval  60  days  prior  to  formal  testing.  They  will 
identify  the  specification  requirement  relating  to  each  test  sequence  and  will 
clearly  state  the  test  criteria. 

3.2  SYSTEM  TEST  REPORT 

The  System  Test  Report  will  follow  the  form  of  the  System  Test  Procedures 
and  will  provide  a  display  of  test  results.  The  criteria  will  be  specified  for 
each  test  sequence  and  the  test  results  will  be  recorded  in  a  form  which  will 
allow  for  ease  of  comparison  to  criteria.  The  test  results  will  be  documented 
in  report  format  and  will  be  combined  with  other  system  assessment  information 
by  the  Training  Analysis  and  Evaluation  Group.  The  System  Test  Report  will  then 
be  published  for  distribution  as  directed  by  the  CNET  Assistant  Chief  of  Staff 
for  Training  System  Management. 
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